Today, IT operations teams have to process large volumes of events or alarms in near real-time in order to protect service levels, stay competitive, and deliver a great experience to customers.
If it takes too long for teams to spot and repair issues, an organization runs the risk of significant business service downtime, SLA penalties, and brand reputation damages. As IT landscapes continue to grow in scale and complexity, guarding against these risks becomes increasingly difficult. To adapt, IT operations teams need to modernize incident management and introduce artificial intelligence (AI) and machine learning to analyze thousands of events in real time.
DX Operational Intelligence enables teams to have a unified view of their monitoring environment and effectively scale to manage all their domain alarms. In this blog, we introduce the concept of alarm clustering and share a few best practices to help you efficiently manage your event/alarm traffic.
What is a Situation Alarm
Situation alarms are a collection of alarms that are correlated and grouped into clusters. Alarms are clustered by combining the alerts based on distinct dimensions. These clusters represent a problem that affects applications, infrastructure, or data center health. This clustering supports simplified triage and more efficient analysis. This grouping also enables you to consolidate a massive number of contextually relevant alarms and address them as a whole. For example, a collection of cascading alarms that are caused by a shortage of specific resources are automatically grouped into a single situation cluster.
From a technology perspective, situations are created using machine learning-based clustering algorithms that employ time correlation, topological relationship, and natural language processing. Situation alarms can map to a number of dimensions, such as text, active time series, host, and historical time series. Alarms can also be mapped based on the relationship of a configuration item to a business service. All these dimensions can be used to create situation clusters.
Situations help reduce alarm noise and provide a holistic view of a problem across distinct IT silos, helping IT operations triage problems faster.
Figure 1: Example of a situation composed of three sub-clusters.
Best Practices to Get Value from Situation Alarms
To get the most out of situation alarms, it is important to follow these best practices:
Harness Timelines
Leverage the timeline to understand the order of events’ arrival. This section contains useful historical information to understand how a situation has developed over time.
Employ Filtering
Build situation filters to focus on what is most relevant for your team. You can zoom in on situations affecting a specific business service or situations in which the root cause has been identified. This is the first step toward creating a policy for ITSM integration or triggering automations from clustered alarms.
Customize Cluster Dimensions
Leverage APIs for clustering situation dimensions to fine-tune how situations are built. You can make adjustments to the weight of each dimension (service, host, time, text, and historical). You can also act on situations via APIs to facilitate automated calls or scripts.
Track Situation Flow
Consider the situation flow. This will help your team better understand which problems should be addressed, and in which sequence. For example, while low entropy problems (that is those that have a low level of change) may be candidates for immediate investigation, problems that have just started may only need to be monitored.
Situations evolve from active, to stable, then to closed.
- Active: New/updated alarms are still forming the cluster.
- Stable (No associated icon): No new or updated alarms have been received.
A situation will transition from active to stable in two possible scenarios:
- Natural stabilization: If no new/updated alarms are received for 30 minutes.
- Forced stabilization: After a configurable “situation window” has passed, this option can force stabilization, even though the cluster may have facets that keep changing.
In addition, these two states (active or stable) can be tagged as “noisy” if new/updated alarms are still arriving in the cluster.
Determine What to Address First
Start by looking at the situations with high severity that are having an impact on your key services or entities. Then, you can decide to tackle situations that are active (still evolving but showing near real-time issues) or stable (cluster already formed and closed).
Leverage Noise Reduction
Leverage noise reduction indicators. This is a great resource to understand how situation alarms are helping your team and improving staff productivity by reducing the number of alarms they need to analyze.
As shown in the example below, instead of browsing through 621 raw alarms, you may only need to handle 12 situations.
Next Steps
Now that you have learned the best practices to deal with alarm flooding by leveraging alarm clustering, get started by exploring our Broadcom Enterprise Software Academy for more DX Operational Intelligence resources or check out this blog: Reduce Noise and Speed Root Cause Analysis with Alarm Analytics