Today, IT operations teams have to process large volumes of events or alarms in near real-time in order to protect service levels, stay competitive, and deliver a great experience to customers.
If it takes too long for teams to spot and repair issues, an organization runs the risk of significant business service downtime, SLA penalties, and brand reputation damages. As IT landscapes continue to grow in scale and complexity, guarding against these risks becomes increasingly difficult. To adapt, IT operations teams need to modernize incident management and introduce artificial intelligence (AI) and machine learning to analyze thousands of events in real time.
DX Operational Intelligence enables teams to have a unified view of their monitoring environment and effectively scale to manage all their domain alarms. In this blog, we introduce the concept of alarm clustering and share a few best practices to help you efficiently manage your event/alarm traffic.
What is a Situation Alarm
Situation alarms are a collection of alarms that are correlated and grouped into clusters. Alarms are clustered by combining the alerts based on distinct dimensions. These clusters represent a problem that affects applications, infrastructure, or data center health. This clustering supports simplified triage and more efficient analysis. This grouping also enables you to consolidate a massive number of contextually relevant alarms and address them as a whole. For example, a collection of cascading alarms that are caused by a shortage of specific resources are automatically grouped into a single situation cluster.
From a technology perspective, situations are created using machine learning-based clustering algorithms that employ time correlation, topological relationship, and natural language processing. Situation alarms can map to a number of dimensions, such as text, active time series, host, and historical time series. Alarms can also be mapped based on the relationship of a configuration item to a business service. All these dimensions can be used to create situation clusters.
Situations help reduce alarm noise and provide a holistic view of a problem across distinct IT silos, helping IT operations triage problems faster.
Figure 1: Example of a situation composed of three sub-clusters.
Best Practices to Get Value from Situation Alarms
To get the most out of situation alarms, it is important to follow these best practices:
Harness Timelines
Leverage the timeline to understand the order of events’ arrival. This section contains useful historical information to understand how a situation has developed over time.
Employ Filtering
Build situation filters to focus on what is most relevant for your team. You can zoom in on situations affecting a specific business service or situations in which the root cause has been identified. This is the first step toward creating a policy for ITSM integration or triggering automations from clustered alarms.
Customize Cluster Dimensions
Leverage APIs for clustering situation dimensions to fine-tune how situations are built. You can make adjustments to the weight of each dimension (service, host, time, text, and historical). You can also act on situations via APIs to facilitate automated calls or scripts.
Track Situation Flow
Consider the situation flow. This will help your team better understand which problems should be addressed, and in which sequence. For example, while low entropy problems (that is those that have a low level of change) may be candidates for immediate investigation, problems that have just started may only need to be monitored.
Situations evolve from active, to stable, then to closed.
- Active: New/updated alarms are still forming the cluster.
- Stable (No associated icon): No new or updated alarms have been received.
A situation will transition from active to stable in two possible scenarios:
- Natural stabilization: If no new/updated alarms are received for 30 minutes.
- Forced stabilization: After a configurable “situation window” has passed, this option can force stabilization, even though the cluster may have facets that keep changing.
In addition, these two states (active or stable) can be tagged as “noisy” if new/updated alarms are still arriving in the cluster.
Determine What to Address First
Start by looking at the situations with high severity that are having an impact on your key services or entities. Then, you can decide to tackle situations that are active (still evolving but showing near real-time issues) or stable (cluster already formed and closed).
Leverage Noise Reduction
Leverage noise reduction indicators. This is a great resource to understand how situation alarms are helping your team and improving staff productivity by reducing the number of alarms they need to analyze.
As shown in the example below, instead of browsing through 621 raw alarms, you may only need to handle 12 situations.
Next Steps
Now that you have learned the best practices to deal with alarm flooding by leveraging alarm clustering, get started by exploring our Broadcom Enterprise Software Academy for more DX Operational Intelligence resources or check out this blog: Reduce Noise and Speed Root Cause Analysis with Alarm Analytics
Nestor Falcon Gonzalez
Nestor holds a Master's Degree in Telecommunication Engineering and has over 20 years of experience in Telco, Network and Infrastructure Operations in different roles: SWAT, pre-sales and Solution Architect. He focuses on helping customers on their network transformation, driving innovation and providing value for...
Other resources you might be interested in
Nobody Cares About Your MTTR
This post outlines why IT metrics like MTTR are irrelevant to business leaders, and it emphasizes that IT teams need network observability to bridge this gap.
Tag(ging)—You’re It: How to Leverage AppNeta Monitoring Data for Maximum Insights
Find out about tagging capabilities in AppNeta. Get strategies for making the most of tagging and see how it can be a game-changer for your operations teams.
Rally Office Hours: October 2, 2025
The Rally Model Context Protocol (MCP) Server acts as a standardized interface for AI models and developer tools. Learn about this exciting new feature then follow the weekly Q&A session with Rally...
Why 1% Packet Loss Is the New 100% Outage
In an era of real-time apps and multiple clouds, the old rules about 'acceptable' network errors no longer apply. See why you need end-to-end observability.
Rally Office Hours: September 25, 2025
Rally Office Hours delivers an essential product tip: Learn to transition from Legacy Custom Pages to powerful Custom Views. Plus, Q&A insights.
Defining the Network Engineer of Tomorrow
Read this post and see why the most important investment isn't in new hardware, but in transforming your team from device managers to service delivery experts.
Harnessing AppNeta’s Browser- and HTTP-based Workflows to Track User Experience
AppNeta’s browser- and HTTP-based workflows let you see what users actually experience. Preempt issues before they become headaches for your end users.
“Rego U” Recap: Why SPM Is Still Hot
Rego Consulting’s Annual Conference underscored why strategic portfolio management (SPM) is still essential. Leverage SPM to bridge strategy and execution.
What's New in AutoSys 24.1: Built for the Modern Automation Landscape
See how AutoSys 24.1 is designed to streamline your daily tasks, accelerate troubleshooting, and simplify how you integrate with the latest technologies.