January 13, 2022
Best Practices for Maximizing the Value of Situation Alarms
Written by: Nestor Falcon Gonzalez
Today, IT operations teams have to process large volumes of events or alarms in near real-time in order to protect service levels, stay competitive, and deliver a great experience to customers.
If it takes too long for teams to spot and repair issues, an organization runs the risk of significant business service downtime, SLA penalties, and brand reputation damages. As IT landscapes continue to grow in scale and complexity, guarding against these risks becomes increasingly difficult. To adapt, IT operations teams need to modernize incident management and introduce artificial intelligence (AI) and machine learning to analyze thousands of events in real time.
DX Operational Intelligence enables teams to have a unified view of their monitoring environment and effectively scale to manage all their domain alarms. In this blog, we introduce the concept of alarm clustering and share a few best practices to help you efficiently manage your event/alarm traffic.
What is a Situation Alarm
Situation alarms are a collection of alarms that are correlated and grouped into clusters. Alarms are clustered by combining the alerts based on distinct dimensions. These clusters represent a problem that affects applications, infrastructure, or data center health. This clustering supports simplified triage and more efficient analysis. This grouping also enables you to consolidate a massive number of contextually relevant alarms and address them as a whole. For example, a collection of cascading alarms that are caused by a shortage of specific resources are automatically grouped into a single situation cluster.
From a technology perspective, situations are created using machine learning-based clustering algorithms that employ time correlation, topological relationship, and natural language processing. Situation alarms can map to a number of dimensions, such as text, active time series, host, and historical time series. Alarms can also be mapped based on the relationship of a configuration item to a business service. All these dimensions can be used to create situation clusters.
Situations help reduce alarm noise and provide a holistic view of a problem across distinct IT silos, helping IT operations triage problems faster.
Figure 1: Example of a situation composed of three sub-clusters.
Best Practices to Get Value from Situation Alarms
To get the most out of situation alarms, it is important to follow these best practices:
Harness Timelines
Leverage the timeline to understand the order of events’ arrival. This section contains useful historical information to understand how a situation has developed over time.
Employ Filtering
Build situation filters to focus on what is most relevant for your team. You can zoom in on situations affecting a specific business service or situations in which the root cause has been identified. This is the first step toward creating a policy for ITSM integration or triggering automations from clustered alarms.
Customize Cluster Dimensions
Leverage APIs for clustering situation dimensions to fine-tune how situations are built. You can make adjustments to the weight of each dimension (service, host, time, text, and historical). You can also act on situations via APIs to facilitate automated calls or scripts.
Track Situation Flow
Consider the situation flow. This will help your team better understand which problems should be addressed, and in which sequence. For example, while low entropy problems (that is those that have a low level of change) may be candidates for immediate investigation, problems that have just started may only need to be monitored.
Situations evolve from active, to stable, then to closed.
- Active: New/updated alarms are still forming the cluster.
- Stable (No associated icon): No new or updated alarms have been received.
A situation will transition from active to stable in two possible scenarios:
- Natural stabilization: If no new/updated alarms are received for 30 minutes.
- Forced stabilization: After a configurable “situation window” has passed, this option can force stabilization, even though the cluster may have facets that keep changing.
In addition, these two states (active or stable) can be tagged as “noisy” if new/updated alarms are still arriving in the cluster.
Determine What to Address First
Start by looking at the situations with high severity that are having an impact on your key services or entities. Then, you can decide to tackle situations that are active (still evolving but showing near real-time issues) or stable (cluster already formed and closed).
Leverage Noise Reduction
Leverage noise reduction indicators. This is a great resource to understand how situation alarms are helping your team and improving staff productivity by reducing the number of alarms they need to analyze.
As shown in the example below, instead of browsing through 621 raw alarms, you may only need to handle 12 situations.
Next Steps
Now that you have learned the best practices to deal with alarm flooding by leveraging alarm clustering, get started by exploring our Broadcom Enterprise Software Academy for more DX Operational Intelligence resources or check out this blog: Reduce Noise and Speed Root Cause Analysis with Alarm Analytics
Nestor Falcon Gonzalez
Nestor Falcon Gonzalez is a Senior Solution Architect focusing on AIOps solutions. He works in the Customer Success SWAT team, providing technical expertise during sales, POC, and implementation cycles and enabling customers to embrace their Digital Transformation successfully. Nestor holds a Master's Degree in...
Other posts you might be interested in
Explore the Catalog
May 11, 2023
IT Operations in 2023: AI/ML & Automation Will Continue to Be the North Star
Read More
May 1, 2023
Enhanced Security, Improved Performance Reporting with DX UIM 20.4 Cumulative Update 7
Read More
April 19, 2023
IT Operations in 2023: Business Services Become a Viable Organizing Principle
Read More
April 13, 2023
OpenTracing via Jaeger
Read More
April 10, 2023
Top Reasons to Embrace a Hybrid Multi-Cloud Strategy
Read More
March 27, 2023
Digital Transformation Drives a New Cloud Era
Read More
March 21, 2023
IT Operations in 2023: Mapping IT to Business Goals
Read More
February 16, 2023
DX UIM 20.4 CU6: What’s New and Why You Should Upgrade Now
Read More
November 1, 2022