In my previous blog, I discussed how continuous observability can be used to deliver continuous reliability. We also discussed the problem of high change failure rates in most enterprises, and how teams fail to proactively address failure risk before changes go into production.
This is because manual assessment of change risk is both labor intensive and time consuming, and often contributes to deployment and release delays. For example, in a large financial services enterprise, change approval lead time averaged about 3.5 days with more than seven approvers involved.
Traditional processes are not only time consuming, they can also be unreliable. That is because they often rely on static algorithmic techniques that are not customized to the unique characteristics of each application or system—and it is laborious to manually develop unique algorithms for each different system.
Observability solutions are key to addressing this problem. Generally speaking, “observability” solutions in the industry tend to focus on operational data. However, I’d contend that teams need to shift observability left. By bringing observability into all stages of the CI/CD lifecycle, teams can glean insights from pre-production data, in conjunction with production data. Pre-production environments are more data rich than production environments (in terms of variety), however, most of this data is not correlated and mined for insights.
For example, it is possible to obtain reliability insights from patterns of code changes in source code management systems. Teams can also gain insights from test results and performance monitoring. This data can be correlated with past failure and reliability history from production.
In this blog, I will discuss more about how we can better predict risks associated with changes, and improve both reliability and overall quality of software systems by employing observability with pre-production data.
Continuous observability in the CI/CD pipeline requires us to setup “systems of intelligence” (SOI, see figure below). SOIs are where we continuously collect and analyze pre-production data along the CI/CD lifecycle. Pre-production data is then correlated with production data to generate a variety of insights when applications change.
Data can be gathered from a variety of tools that form a part of typical DevOps pipelines. This can include the following:
For purposes of reliability and quality analytics, machine-learning-based models are built by correlating pre-production data with problems that have arisen in past production deployments. This can include system failures, escaped high severity defects, service degradation incidents, deployment failures, customer experience issues, and SLA violations.
In my experience, I find that a generic machine learning model is not well suited for all applications. It is best to build a model that’s aligned with the specific characteristics of the application.
Note that in some cases, it may not be possible to construct a valid, accurate model that aligns with performance criteria. For example, this may be due to the application data being captured. In these cases, it would not be possible to use the mechanism outlined above to make accurate predictions. Teams must have suitable data that enables effective model generation.
Once the machine learning model is created with past pre-production data, teams can use this model and data to predict the deployment risk for new changes.
Risks may be predicted at various levels of product or software granularity:
In the previous section, I discussed risk prediction at different levels of application software granularity. In this section, I’ll outline how we make the risk prediction process “continuous” and tie it to the different phases of the CI/CD lifecycle. In a continuous risk prediction system, the risk score is updated continuously as the software proceeds from the left to the right of the lifecycle—from development to production.
At every stage, we understand the risk factors, and address them continuously to progressively reduce the risk until the software gets to production. In each phase, risk prediction helps to provide guidance—in near real time—on whether the build should be promoted or not. This decision can either be made manually or automatically by pushing the data to a decision system. Even more importantly, this also provides insights on how to address risks.
This mechanism enhances the continuous feedback process with precise, data-driven insights, which are so important in DevOps.
Let’s look at how this works (see figure below).
The periodic update of the prediction model using the latest data is another key nuance to the continuous approach to risk prediction, which ensures that the prediction model automatically reflects the characteristics of the application as it evolves over time.
The mechanism described above is shown for an individual pipeline, such as a microservice or application component. When multiple pipelines are part of a unified release (such as in an Agile Release Train in SAFe), the risk score for the entire ART is computed based on the risk scores of the individual pipelines.
The key overall benefit of continuous risk prediction is to reduce change failure rate and to reduce manual overhead associated with change approvals in a CI/CD pipeline.
Continuous risk prediction benefits a variety of personas, as described below.
While there are specific benefits for these personas, the key unstated benefit is the data-driven cross-persona collaboration that it enables across the lifecycle. Many of these personas often work in the context of their specific tools, processes, and metrics.
Despite the availability of DevOps dashboard tools (like Hygieia) it is sometimes challenging to enable collaboration across silos to enable decision making. That’s because these tools do not enable teams to correlate data across the various sources they are drawing from. This leads me to the final point I want to make about the difference between just having data versus having data that provides actionable insights.
DORA metrics provide great indicators of the performance of an application team. However, such metrics are good for descriptive analytics, that is, they describe “what is happening now.”
A continuous observability solution like the one we describe in this blog goes beyond descriptive analytics to provide valuable insights into “why did it happen.” (This can include diagnostic analytics.) For example, this can help teams understand “why do we have such a high change-failure rate?”
And further, they provide even more valuable predictive and prescriptive analytics, offering insights into “what changes do I need to make and what impact will they have?”
The latter are keys for driving better continuous improvement in DevOps. The “Third Way of DevOps” is about creating a culture of continual experimentation and learning. Continuous observability solutions provide a structured, data-driven approach that supports this cultural shift.
This blog has described how teams can use observability-based, continuous risk prediction to reduce change failure rates.
However, change failure is just one dimension of application and system quality. For example, it does not fully address how we optimize customer experiences, which are influenced by other factors, such as ease of use and value. In order to get true value from DevOps and continuous delivery initiatives, we need to establish practices for predictively attaining quality—we call this “continuous quality.” I will discuss this in my next blog.