Learn how Automic can help NetOps speed up triage, reduce MTTR, and alleviate the load on your NOC
Today’s network team is committed to advancing the speed of triage and resolution. This can be a titanic task in modern networks, which are fueled by SDN and becoming more agile and dynamic than ever.
These new networks have introduced traffic, congestion, and outages on a greater scale. We hear constantly from organizations about not being able to deal with the amount of tickets and events raised in their NOCs. This level of demand has elevated Network Automation to an unprecedented level: 49% of enterprises consider automation a major priority in the next one to two years1.
Leveraging automation of SOPs (Standard Operating Procedures) from NetOps will help your team spend less time in triage and more time focusing on what matters. Today, 43% of network professionals are challenged to find the time to work on strategic business initiatives2.
This skill builder blog will elaborate on a key capability of the Broadcom NetOps solution: leveraging Automic to trigger workflows on common repetitive tasks executed by your NOC Team. Utilizing this feature saves triage time and automates SOPs, delivering more efficient results for your organization..
It is important to note that Network Automation without validation is a high risk operation, so any SOP automatically executed must be approved and validated within your organization.
You will start by enabling NetOps as the single pane of glass to drive triage from. This is achieved by configuring the DX Spectrum OneClick console with a new alarm menu item; from here, operators can trigger Automic workflows.
Figure 1: Enabling custom menu picks in DX Spectrum OneClick
This task will save a considerable amount of NOC users’ time. DX NetOps users can now invoke actions in context from a given alarm. This action can take any field from the selected alarm and consume it as a part of a workflow or command (e.g., you can use the “alarm Title” and “Type” fields to run a search in the KB and enrich Alarm Notes with this information).
Note that security is heavily embedded in our NetOps solution; admins can associate a privilege to the menu item, so if the user is not given this privilege, the menu item is not displayed for that user.
In the current scenario, we have simply executed a script that will invoke the Automic workflow via API. This action will use two alarm fields as parameters. Below is an extract from the file <$SPECROOT>/custom/console/config/custom-menu-config.xml.
<menu name="com.aprisma.spectrum.app.swing.window.menu.Tools">
<item name="Automic Workflow Healthcheck">
<toolbar-image>images/bclogo.jpg</toolbar-image>
<popup-visibility>when-filter-passed</popup-visibility>
<action>
<context>
com.aprisma.spectrum.app.alarm.client.group.AlarmContext
</context>
<filter>
<has-attribute>AlarmAttrID.ALARM_ID</has-attribute>
</filter>
<launch-web-server-script>
<command>/usr/automic-script/workflow-healthcheck.sh {0} {1}</command>
<param>
<attribute>AlarmAttrID.ALARM_ID</attribute>
</param>
<param>
<attribute>AttributeID.MODEL_NAME</attribute>
</param>
</launch-web-server-script>
</action>
</item>
</menu>
Beyond just launching this action manually, you could leverage policies and AlarmNotifier to execute some of these tasks in an automated manner.
Next, define the workflow to be executed in Automic. There are some examples in the last part of this document, but in our scenario, we will just run a healthcheck on the alarmed network entity and enrich the NetOps alarm so operators are aware that the triage for that alarm has been completed successfully.
The Automic workflow is invoked via a REST API call, and some alarm fields are passed along for processing. Depending on the task to be performed, you will need to design your own workflow or leverage one from the Automic Marketplace.
Figure 2: Invoked Automic Workflow to run a healthcheck and update Spectrum alarm
It is important that you close the loop to let NOC operators know the outcome of the automated action. You can achieve this by simply updating the Spectrum alarm via API call with the output of the automic job and a hyperlink in case additional information is needed.
NOC operators can now continue with the SOP by opening a ticket that will be enriched with the data from the healthcheck executed.
Figure 3: DX NetOps alarm enriched with Healthcheck outcome and workflow status hyperlink.
As you can see, it took a single click for the operator to trigger a healthcheck workflow and enrich the NetOps alarm with valuable information on the outcome. These insights optimize operations and directly impact MTTR.
Some common SOPs that we often hear about and can be automated are the following (Note that they can all be driven with read-only procedures to mitigate any security concerns):
It is important to start with “low hanging fruit” that can provide huge time savings with little effort. A best practice would be to get a report of the most common alarms and most repetitive tasks performed by the NOC.
Based on the maturity level of each organization, we can build a framework for manual tasks, then constantly assess, improve, and validate automations. Lastly, we then aim for closed-loop remediations.
1BCG Covid-19 IT Buyers Sentiment Survey, April 24-May 5, 2020
2Sirkin Research and LiveAction study to identify the top network performance challenges. https://www.liveaction.com/resources/blog/2019-top-network-performance-challenges/