Power Up Your Alarms! Enriched UIM Alarms for Added Intelligence

The use case

An often-overlooked, powerful feature of DX UIM (Unified Infrastructure Management) is the Alarm Enrichment probe. Deployed on the Primary Hub as part of the standard installation, this feature has significant, often untapped potential to enhance the effectiveness of alarms generated by DX UIM.

Without configuration, alarms pass through the Alarm Enrichment probe and are processed by the NAS (Notification and Alerting Service). However, if configured properly, user-specific data can be added to the alarms before they are processed. Generally, alarms created by DX UIM policies are informative; they indicate severity and source, while the message conveys what the monitoring has detected. The key term here is generic—the alarms must meet the needs of a broad customer base. Yet, each customer has unique requirements, processes, and downstream systems. This is where Alarm Enrichment becomes essential for modifying and augmenting alarm information to increase its effectiveness.

NAS pre processing

The simplest form of enrichment occurs in the NAS. If you want to add fixed data to alarms, the NAS offers pre-processor profiles that can manipulate messages (or other fields) based on user-defined scripts. This represents enrichment in its most basic form and is useful for adding prefixes or suffixes to alarm messages. However, the scope of added data is limited to what is hard-coded in the NAS script.

Pre-processing happens before the alarm is inserted into the database. For instance, one practical application is adding DX UIM severity to messages when downstream systems only recognize three severities instead of the solution's five.

To implement this in the Script tab of NAS/Auto Operator configuration (available only in Infrastructure Manager):

1. Right-click and select New Script.
   
2. Create the following LUA script:
   
3. Create the following Pre-Processing Profile:
   

As per the above profile, all Warning (level 2) alarms from the cdm probe will be prefixed with WARNING:

Note 1: You can edit these fields in an inbound alarm using a pre-process script:

• Message
• Level
• Sid
• Source
• Hostname
• User_tag1
• User_tag2
• Visible
• Custom_1 to Custom_5
• Supp_key
• Origin

Note 2: The following fields can be used alongside those in Note 1 but cannot be modified:

• Domain
• Robot
• Hub
• Prid

NAS Alarm Enrichment

If the data required for enrichment is more extensive and varies based on configuration items, it should ideally be stored in a database table or would need to be created. In such cases, the Alarm Enrichment probe should be utilized. This probe reads alarms off the message bus and queries a database based on selection criteria to enrich them accordingly.

The added data might include links to knowledge bases, group email addresses for support teams, or group IDs used by downstream systems. Typically, this data will be placed in one of the custom fields that are empty on inbound alarms but populated during processing by Alarm Enrichment/NAS.

Note: This probe is unique as it isn’t configured through its local .cfg file; instead, it relies on keys in NAS.cfg within specific sections:

This configuration can only be achieved using the Raw configure in either Infrastructure Manager or Admin Console. Alternatively, you can edit the NAS.cfg directly on the Primary Hub, following with a restart to re-read the cfg file.

Enrichment-source section

Here, data sources and queries are configured along with connection strings to databases and credentials.

Connection_url—The data may not yet exist, therefore the implementor can point to a remote database or a newly created table in the DX UIM database.

The Population Query caches data from this table; subsequently, a specific Query retrieves data for actual configuration items (CIs).

The Query is used to select a single row of data, which is to be used to insert into the Alarm. The ‘?’ should be substituted with what as been defined in the lookup_by_alarm_field key.

The password field is encrypted when the NAS restarts.

Enrichment-rules section

This section consolidates everything. The Use Enricher key specifies which database and query to use for data retrieval as defined in the Source section.

Match_alarm_field indicates which field to select alarms on—typically Prid (probe).

Match_alarm_regex defines regex patterns for selecting multiple values.

Lookup_by_alarm_field specifies which field will have its values substituted in queries for enrichment.

The [cmdb.*] are the fields retrieved from the database lookup and, in the above example, are placed in the matching custom fields in the alarm. The fields may be overwritten but be careful not to lose alarm key data.

Summary

By implementing these configurations, selected alarms will contain additional data within their custom fields. This enhancement can significantly aid operators by providing action descriptions that facilitate quicker remediation efforts, ultimately reducing mean-time-to-recovery (MTTR) statistics.

The above configuration outlines a single enrichment process; however, multiple enrichment rules can be executed on the same alarms in sequential order as defined in the cfg file, allowing for comprehensive enrichment capabilities across various alarm scenarios.