June 17, 2025
Reduce Alarm Noise, Improve Triage, and Route Tickets to the Right Team the First Time
6 min read
|
Key Takeaways
|
|
IT operations teams in enterprises across industries and of all sizes share similar goals: offsetting the challenges of “too many alarms.” This includes addressing these objectives:
- Get alarms to the right teams the first time. Include meaningful context.
- Avoid sending false alerts that are unwarranted to SME teams.
- Cluster related alarms.
- Don’t overlook important alarms (true signals) as you strive to achieve the other goals above.
During a recent conversation with a large financial services customer, people I spoke to recounted the challenges their teams faced with managing so many alarms. Level-1 teams were getting alarm fatigue. At the same time, understanding and segregating meaningful signals from noise was an ongoing challenge. Worried about missing important, genuine issues, teams would err on the side of over-engaging SMEs (including on false alerts), which created additional work for all involved. The result: without a better approach to alarm management, operational responsiveness and SLA/SLO attainment would suffer.
With increasing adoption of cloud services, containers, and microservices, the IT landscape has become significantly more complex. This customer turned to DX Operational Observability (DX O2), and the Situations capability to address these challenges. Situations can help teams programmatically and consistently address these questions:
- Which alarms are noise? Which are signals?
- Which alarms are related to other alarms?
- Which should take priority for triaging?
- What is the root cause and which teams should be notified to begin remediation?
- What contextual information is available that can be appended to the ticket to speed remediation?
By providing answers to these questions, DX O2 can help improve key business metrics like mean time to resolution (MTTR) and mean time to innocence (MTTI) and highly skilled experts can focus on innovative work.
“One of the top priorities for CIOs is staying ahead of emerging technologies and solutions.”
—Deloitte (Based on February 2024 poll of 211 US-based CIOs [Source: CIO.com, “The 10 biggest issues IT faces today”])
Achieving alarm noise reduction and improving alarm routing is made possible by combining rich observability with powerful AIOps. Let me elaborate with two specific examples.
Alarm noise reduction
This is a powerful capability of DX O2, which is delivered in part through the Situations feature. With Situations, users can reduce the alarm set by configuring a high-level rule for alarm filtering. Using this rule, DX O2 will analyze alarms based on attributes such as message text, entity, device, and business service to determine relationships between alarms and cluster those that are indeed related. This gives users tremendous flexibility for reducing alarm noise, without risking loss of signal.
With Situations, users can also isolate potential culprits, the potential root cause. This helps expedite routing incidents to the right teams.
Using Situations in DX O2, a large health insurance provider customer achieved noise reduction of 98.6%. This improves both the quality of results and the efficiency of teams across IT. Before adopting DX O2 and Situations, the customer noted, “Looking for the right alarm in the flood of alarms was like looking for important rain drops in a hurricane.” Using DX O2, alarm fatigue is reduced, important alarms get proper attention, and teams can focus more on actions instead of sifting through mountains of information.
Alarm routing
In broad terms, alarm routing is a matter of triaging the right alarm and getting it to the right team on time. This means that alarms need to have relevant information, that is, sufficient context so they can be routed via the IT service management (ITSM) system to the right teams. Relevant information likely includes details on impacted CIs, the business name of the application, and so on. If the information is not available natively, then the platform should enable users to enrich the CI and the alarms with appropriate attributes. Once the enriched alarm is dispatched, the ITSM system can then route the event based on the additional details.
In another case, a customer using DX O2 wanted to route alarms through the ITSM event module to appropriate teams. The customer enriched the CIs using a key attribute ingested into DX O2 from their CMDB. Enriching tickets with attributes such as the name of the business service being monitored is a golden ticket: It informs IT operations which team owns the service and how to notify that team. For this customer, the business service name is critical for triaging associated events and getting them to the right team. Alarm enrichment using DX O2 allows their teams to enrich alarms with associated CI details so that events sent to ITSM include crucial business context. In addition, this helped ensure alarm routing was accurate, timely, and precise.
“CIOs face immense pressure to deliver successful digital initiatives while navigating budget constraints and increasing demands from senior executives.”
—Gartner, “Priorities CIOs Must Address in 2025, According to Gartner’s CIO Survey”
By reducing alarm noise and efficiently routing alarms to the right teams the first time—and with helpful context—teams can more confidently adopt cloud services, containers, and microservices, while ensuring the business services these environments support are healthy and performing well. When issues arise, they can be addressed quickly and with less effort, creating benefits for all of IT.
Srikant Noorani
Srikant Noorani, Client Services Architect focusing on AIOps and Observability, has over 20 years experience working on complex technical challenges. A hands-on architect with a passion for guiding enterprises in their digital transformation journey, Srikant has worked on the largest APM deployments plus DevOps,...
Other Resources You might be interested In
Observability and Monitoring Governance (Part 1 of 4)
Find out how strong monitoring governance can help IT teams cut through the noise, see what truly matters, and act with precision.
Observability and Monitoring Governance (Part 2 of 4)
Read this post and discover some of the top downstream benefits of establishing strong monitoring governance. Gain best practices on how and where to start.
DX UIM Hub Interconnectivity and the Benefits of Static Hubs
Find out how using static hubs is a powerful way to enhance observability. Discover when and how to use static hubs, and the benefits they can provide.
Broadcom Recognized as a Leader: Engineering the Future of Service Orchestration
Read this post and see why Broadcom was named a Leader in the 2025 Gartner® Magic Quadrant™ for Service Orchestration and Automation Platforms.
Customer Spotlight: Global Bank MUFG Saves Millions of Dollars
MUFG’s Bruce Frank discusses how the global bank invokes Broadcom's Automated Analytics & Intelligence (AAI) to manage SLAs and ensure regulatory compliance, saving millions of dollars annually.
The "Lighthouse" of Strategy: Guiding Your Organization Through Decision Chaos
Strategic clarity is key. See how strategic portfolio management (SPM) helps align resources and decisions for better business outcomes and ROI.
4 Ways AppNeta Enhances Cost-Focused Cloud Planning
See how AppNeta delivers insights that enable cloud architects to correlate wasted spending with performance degradation and proactively relocate resources.
Automic Automation Cloud Integrations: Azure Functions Agent Integration
Broadcom's Azure Functions Automation Agent lets you easily execute Azure Functions, monitor and manage them with your existing enterprise workload automation, as well as other cloud-native...
The Public Internet Is Not Your WAN
Moving beyond MPLS was a strategic necessity. To succeed in modern environments, you need to stop guessing about internet performance and start measuring it.