DX Operational Observability: Troubleshoot WebHook Notification Channels with WebHook Data Collector

Introduction

The power of AIOps and Observability relies on the ability to ingest, normalize, and correlate the large volumes and huge variety of data available to IT operations teams. With its support for both Broadcom and third-party data, DX Operational Observability (DX O2) gives these teams unmatched observability and insights.

With so much data coming to DX O2, monitoring operators need to be notified when important events may occur:

• “Unable to reach WebHook target.”
• “Wrong token due to failed charset.”
• “Wrong token used for authentication.”
• “Which payload is my alert sending over?”

Without notifications, important alerts may be overlooked. By enabling and configuring WebHook notification channels, teams benefit from automation and policy definitions that can be precisely tailored to specific needs.

Unfortunately, these WebHook receivers are limited in their ability to help teams troubleshoot issues that periodically arise with incoming data. The WebHook Data Collector (WHDC) is a small application that addresses this gap. It allows the operator to validate the communication link, the sent payload (including full headers), and GET/POST data if any.

The WHDC is typically deployed on an existing, publicly accessible Kubernetes cluster. It can also be deployed on-premises. When used with DX 02 SaaS, WHDC requires a public, fully qualified domain name (FQDN) and firewall rule-tuning to let the request get through to the deployed WHDC. Note:

• When deploying the WHDC, make sure you have a company or publicly signed certificate, as self-signed certificates will not be accepted due to security reasons.¹

Once the WHDC is deployed, the operator can check the configuration of a WebHook notification, including the payload, headers, and GET/POST that have been sent.

Within DX O2, you can test the regular WebHook channel. You can also test Slack and PagerDuty notifications.

Using WHDC: An example

To illustrate by example, let’s create a notification channel called “Test.”

WHDC requires the tenant-name+test and an e-mail address to identify the user and the notification channel in logs.

This will create a registered tenant:

In this example, let’s configure the notification to include two entries:

1. View x Rows: This will show how many WebHook requests have been captured.
2. Show Token: This specifies the token² that will be used for the WebHook channel configuration and as authentication for viewing results.

Note, for null situations, when there are no rows to display, the access log will be displayed. This helps practitioners identify cases where the wrong token is used.

Below is an example using a wrong token, which is noted in the configuration detail.

Both the DX O2 configuration interface and the WebHook receiver site will show that the token is not correct. Showing the issues on both sides helps to ensure issues are not overlooked. Going further, teams can run a test which, in this example, displays a failure showing that the WebHook authentication failed.

Teams can also view detailed logs on the WHDC target.

In this example, the log reveals helpful details:

1. The log confirms the FQDN (target) is correct: The SaaS tenant can reach the WHDC target.
2. The log however shows the authentication token is incorrect.
3. Also, the token field does not have the Bearer tag.

To resolve the two issues, copy the token from the whdctoken.php interface.

Insert this into the WebHook channel configuration field and add the Bearer tag in front if required. Retesting confirms the issues are resolved.

We now have a successful WebHook submission and can view the payload detail.

If the operator needs additional details, selecting the “Headers” link on the lower left will show the full server-side content.

Conclusion

WebHook Data Connector is a troubleshooting tool to capture WebHook requests and to verify that the token and the data sent (including payload and GET/POST) can actually be seen correctly. It can also be used to validate the payload and associated variables (which change depending on the error). Once the submission has been validated, all the operator needs to do is change the target URL and the token to point it to its final destination.

Endnotes

1. With self-signed certificates, the secured communication will not take place because the DX O2 sending side will not trust the receiver.

2. The Token is a regular Jason Web Token (JWT), and is valid for no more than 12 months. Due to the nature of these tokens, every receiver creates its own tokens (private key in cryptography).