January 24, 2022
In my prior blog, Continuous Service Virtualization, Part 1: Introduction and Best Practices, we offered an introduction to continuous service virtualization (SV) and discussed some key best practices. In this, the second and final post in the series, we will discuss the continuous SV lifecycle and how it helps to optimize DevOps and the continuous integration/continuous delivery (CI/CD) pipeline.
Pulling it All Together: Continuous SV Lifecycle
The following figure summarizes the different activities in a typical continuous SV process across the different stages of the CI/CD pipeline.
Figure 1: Continuous SV lifecycle
These activities are summarized below:
Step 1: Agile Design: Identify Dependencies and Generate a Virtual Service
The SV activities during this stage need to support the requirements of agile development (Step 2). Following are some of the key steps:
- The developer analyzes the dependencies between the components under development and other components.
- The tester or software development engineer in test (SDET) analyzes the needs of various types of tests in the remainder of the CI/CD lifecycle stages. Through this analysis, they determine the right mix of tests, test data, and virtual services needed. This analysis is based on the principle of progressive virtualization discussed above.
- If virtual services for dependent components are already available in the service catalog, they can be marked for re-use. If not, the SDET or developer can generate a synthetic virtual service if the dependent component is not yet implemented. Alternatively, they can record a virtual service if the dependent component is already implemented.
Step 2: Agile Parallel Development and Test
The SV activities at this stage support the needs of the subsequent stages of the CI/CD lifecycle. Following are a few of the most critical efforts:
- The developer or SDET uses the virtual services created in step 1 (above) for unit or component tests.
- In parallel with this development work, testers or SDETs select and enhance the appropriate virtual services with more robust test data and transaction support. These efforts are based on the design analysis done in Step 1(a) above. To support performance tests, virtual services must be configured with appropriate response time settings based on the service level objectives (SLOs) or service level indicators (SLI) of the dependent components.
- The deployment engineer packages those virtual services (along with other test assets) appropriately (in containers or in artifact repositories) so that they can be provisioned and deployed in conjunction with deployment automation tools. This is key for supporting deployment automation in the CI/CD environment. It is important that these test assets, including virtual services, are tagged appropriately for specific application version builds (see next step). They also need to be tagged for the deployment environment so that the right test assets are deployed with the right software builds in the context of specific environments.
Step 3: Commit, Build, and Verification Tests
Typically orchestrated by the CI engine, this is an automated step that triggers the build of the necessary software components and the running of a variety of tests, including build verification or smoke tests, regression, component integration, and so on. This automation uses the virtual services that were created in step 2.
Step 4: Tests Along the CD Lifecycle
The tests along the CD lifecycle (from X-team integration to pre-production tests) use different types of virtual services based on the principle of progressive virtualization. In this way, tests get progressively more realistic as they move towards the right. Also note that the mix of virtual services will be different in each stage of the lifecycle as will the scope of the tests. Another interesting and evolving approach is one in which site reliability engineers (SREs) use service virtualization for performance canary testing, chaos engineering/testing, and other forms of negative testing in production or near-production environments. For example, using virtual services, it is easier to emulate significant service degradation, or even failure, as opposed to adjusting a real service.
How does continuous SV support the needs of DevOps?
Even within traditional COE models, the use of service virtualization provides significant benefits in the context of testing. (These benefits are well documented here.) This includes enablement of agile parallel development and productivity improvements, support for the shift-left of testing, better application quality, easier management of test environments, and more.
Continuous SV, on the other hand, is an integral supporting pillar of continuous testing (CT) and CD (as part of a continuous everything philosophy) and is a key success factor for both. This continuous SV approach offers these key benefits:
It is Essential for Continuous Testing and Continuous Delivery
Progressive virtualization described above is a key tenet for supporting CT/CD. It ensures that appropriate virtual services are created, updated, and re-used in every part of the CI/CD lifecycle, by different stakeholders, and not just during testing. For example, if a virtual service for a dependent endpoint is not already available, developers can create and use lightweight synthetic virtual services for unit and component tests. Testers can then enrich those virtual services with additional transaction support and test data. Another key tenet is the integration with continuous test data management (see my blog on that subject here). Such integration ensures that virtual services can not only help with the creation of test data, but also can be created “fit for purpose” with appropriate test data and transaction support for different types of testing needs. One can argue that for most applications it is not practical to do CT or CD without continuous SV.
Improves Collaboration and Productivity Across Various Stakeholders
Continuous SV requires active participation from, and collaboration between, various federated stakeholders across application teams—including architects, developers, SDETs, testers (both functional and performance), test data engineers, release/deployment engineers, and Site Reliability Engineers (SREs) to name a few. This is different from traditional SV COE models in which virtual services were primarily created and maintained by testers. With limited collaboration, these earlier approaches failed to scale in a rapidly changing application landscape.
Continuous SV ensures that virtual services are an integral part of the agile design and development process. Virtual services are continuously updated (in the context of application changes), re-used, easily deployed with automation, and synchronized with real-world data from production. This active “whole team” collaboration—a key tenet of DevOps—also means that it improves the productivity of all these different stakeholders, not just testers alone.
Easier to Scale Across Multiple Applications to Keep up with Frequent Changes
As a corollary to the traditional COE approaches outlined above, the “whole team” federated approach to continuous SV makes it easier to scale in large enterprises with multiple applications undergoing frequent changes. In this approach, virtual services are treated like first-class “software” objects with a disciplined software engineering approach. For example, virtual services can be tracked and updated using a source code change management process (and versioned in source control repositories) by various stakeholders. They can be packaged for deployment along with applications, and they can be shared and accessed using a centralized service repository. This software engineering (or some call it “as-code”) approach to continuous SV enables better scalability.
Reduces Overall Cycle Time (or Lead Time for Change)
While use of SV helps to make it simpler for testers to do testing, that alone may not significantly impact the overall delivery cycle time. Whereas continuous SV—by virtue of the fact that it helps to accelerate cycle time in every step of the CI/CD lifecycle—has a much more significant impact.
Better Support for Continuous Reliability and Other SRE Initiatives
Continuous reliability is an SRE discipline that seeks to pro-actively assure reliability of applications in the context of frequent changes deployed to production. A continuous SV approach allows us to emulate virtual services more realistically with appropriate SLOs as part of continuous reliability testing across the lifecycle. Where SLOs are not defined (for non-critical components), SREs can obtain SLI data from such components in production and emulate their performance correctly in pre-production. In addition, as mentioned above, SREs take advantage of virtual services to perform negative and chaos testing in production or near-production environments.
How do we get started with continuous SV?
Continuous SV is meant to be practiced in conjunction with CT and CD. Various resources offer insights into evolving to continuous testing. If you are already practicing CT/CD and want to move to continuous SV, our recommendation is to proceed as follows:
- For new functionality, follow the continuous SV approach we have described.
- For existing software, you may choose to focus continuous SV efforts on the most problematic or change-prone application components, since those are the ones you need to test most often. Introduce dependency analysis as part of the agile design process, and aggressively virtualize dependencies on other components.
- For other components that do not change as often, you need to test less often. Therefore, pick and choose the virtual services you need to create as part of your remediation of application technical debt.
Organizational Change Management is Key to Continuous SV Success
Regardless of the approach you take, you need to also make some organizational, cultural, and skills adjustments. These are related to changes you need to make for supporting CT/CD overall. Here are some of the changes that relate specifically to continuous SV:
- Make SV a federated effort by applying a “whole team” approach rather than a traditional COE approach. Various stakeholders in the CI/CD lifecycle need to understand and contribute to continuous SV efforts. Traditional COEs need to evolve to modern centers of enablement that provide support to the application teams in applying continuous SV best practices. In some cases, we see such centers of enablement also own the tooling and infrastructure related to SV.
- Invest in learning new SV skills. Obviously, we need to train stakeholders like developers, deployment engineers, and SREs on SV, but we also need to train testers to work better with developers and other stakeholders. Sometimes, new roles that fuse development and test skills, like SDETs can help as well.
- Take a software engineering (or “as-code”) approach to SV rather than classic test asset management approaches. This is required to ensure collaboration across various stakeholders and achieve scale.
Generally speaking, these organizational and cultural changes are the most difficult aspect of transitioning to continuous SV. Luckily, organizational change management processes in DevOps are now becoming a more established discipline. These same principles will help with continuous SV as well.
Until such time, my friends, stay well, and may all your SV efforts be continuous!