DISCLAIMER: This blog describes a field-developed pilot not officially supported by Broadcom. The vendor does not provide any warranties or support for issues arising from this implementation. |
Key Takeaways
|
|
In today's interconnected world, ensuring network reliability and performance is not just important—it's a must. Network alarms serve as the first line of defense in identifying and mitigating potential issues, providing network operations teams with the actionable insights they need to respond swiftly and effectively. To truly empower network operations teams to boost agility and efficiency, these alarms must be real-time and actionable.
Organizations can significantly enhance their network observability strategy by implementing a streaming network alarm pipeline. This approach offers many benefits, including in-depth performance analysis, rigorous compliance and auditing, and the potential for advanced machine learning and predictive analytics. These streaming pipelines can help with immediate issue detection and forecasting future network anomalies, leading to a more proactive operational stance.
This post will walk you through the process of building a robust streaming data pipeline by using Kafka to send DX NetOps by Broadcom events to Google BigQuery. Once the data is in BigQuery, we’ll explore how to leverage Google Looker to visualize key insights, turning raw network data into actionable intelligence that promotes better decision-making and enhances overall network performance.
Figure 1: Streaming data pipeline components.
We begin by integrating DX NetOps (Spectrum) with Apache Kafka to publish network alarms. DX NetOps offers seamless, out-of-the-box integration with Kafka, making the setup straightforward. The key step is configuring the Kafka alarm producer—a client that operates on a OneClick Server and publishes alarms to the Kafka cluster.
To configure this integration, navigate to the configuration file located on the OneClick server at the following location:
<$SPECROOT>/tomcat/conf/application-spring.properties
Within this file, you'll need to specify the Kafka cluster details, including the broker's IP address, port, and the Kafka topic where the alarms should be published. After entering the necessary information, simply restart the OneClick process, and your alarms will start flowing into the Kafka cluster.
To visualize the DX NetOps alarms being published, you can use any open-source Kafka UI tool. These interfaces will allow you to check alarms published into Kafka messages.
Figure 2: TMF payload for the events.
To learn more, view our technical documentation, which features in-depth information on the Kafka integration.
By integrating DX NetOps with Kafka, you unlock a wide range of powerful use cases. This setup enables the automated creation of service desk tickets directly from Kafka events, streamlining incident management. It also facilitates easy integration with data lakes, such as Elasticsearch or Splunk, allowing for advanced analysis and correlation across various platforms. These capabilities not only enhance your network observability but also pave the way for more efficient operations and data-driven decision-making.
The next step involves determining where to store the DX NetOps alarms within Google Cloud. For this, we'll use Google BigQuery, a fully managed, AI-ready data platform that simplifies complex data analysis and provides powerful capabilities for querying large datasets.
Start by logging into your organization's Google Cloud Platform account. Once logged in, enable the BigQuery service if it's not already active. Next, create a BigQuery dataset and a table within it, following the structure outlined below. This setup will serve as the repository for your DX NetOps alarms, enabling efficient storage, retrieval, and analysis of your network data.
Figure 3: BigQuery dataset and table created for DX NetOps.
To create the BigQuery table, the first step is to define the table schema, which specifies the column names and data types. In BigQuery, a schema is simply a file that outlines these details.
There are several methods available for creating BigQuery schemas. For this pilot project, we will define the schema based on a Spectrum alarm in JSON format. Here’s how to proceed:
schema.json
).bq mk --table netops_dataset.netops_table_alarm schema.json
Your table definition should resemble the example provided below:
Figure 4: Table schema definition in BigQuery.
When considering Google Cloud costs, it's essential to regularly review your Google Cloud Platform pricing details to stay informed. BigQuery offers generous free usage limits: The first 10 GB of data storage and the first 1 TB of processed query data per month come at no cost. This allowance makes it possible to conduct this small-scale pilot without incurring any charges, ensuring you can explore the solution's potential without financial risk.
At this stage, we've successfully set up our DX NetOps alarms within the Kafka cluster and prepared our BigQuery dataset and table. The next step is to connect these two systems, enabling seamless data flow from Kafka to BigQuery.
In Google Cloud Platform, the next step is to create a Pub/Sub topic along with a subscription. This can be done in a single step, as illustrated in Figure 5.
The Pub/Sub topic acts as the entry point for our data into Google Cloud, serving as the bridge between your Kafka data stream and BigQuery. The subscription functions as the mechanism that Google Cloud uses to route data from the Pub/Sub topic to BigQuery, essentially acting like a message bus that ensures the data is delivered accurately and efficiently.
Figure 5: Topic configuration in BigQuery.
Important considerations when creating the Pub/Sub topic:
Now that we have our Pub/Sub topic set up, the next step is to stream DX NetOps alarms from Apache Kafka to this Pub/Sub topic. To achieve this, we’ll use the Pub/Sub Kafka Connector, which acts as a Kafka consumer, reading alarms from the specified DX NetOps Kafka topic and forwarding them to the Pub/Sub topic.
There are two key configuration files you'll need to set up for this connector:
cps-sink-connector.properties
connect-standalone.properties
After configuring these files, the connector will be ready to stream data from Kafka to Pub/Sub.
Finally, ensure you create a Google Service Account within your Google Cloud Platform project and assign it the Pub/Sub Admin role (Figure 6). This is necessary to grant the connector the appropriate permissions to publish messages to the Pub/Sub topic.
Figure 6: Google Service Account creation.
Additionally, create and download a private key for the Kafka connector (Figure 7). This key is required for the connector to authenticate and write data to the Pub/Sub topic.
Figure 7: A key is required for the connector to authenticate.
The connector is now ready to be executed. To streamline this process, wrap the full command in a script (e.g., runconnector.sh) as shown below. Be sure to verify your file paths and execution permissions.
(Note: Ensure that the private key generated earlier is copied locally to the system, as it is required for the connector to authenticate and write to the Pub/Sub topic.)
#!/bin/bash
export GOOGLE_APPLICATION_CREDENTIALS=/root/pubsub-kafka-connector/gcp-nestor-key
/opt/CA/netops-kafka/kafka/bin/connect-standalone.sh \ /root/pubsub-kafka-connector/config/connect-standalone.properties \
/root/pubsub-kafka-connector/config/cps-sink-connector.properties
Once the script is set up, run the connector. You should see it begin to consume messages from Kafka and forward them to Pub/Sub. Subsequently, Pub/Sub will utilize the BigQuery subscription to populate the table we created in BigQuery.
To verify that the DX NetOps alarms are being correctly stored in BigQuery (Figure 8), you can use a query such as:
“SELECT * FROM `nestordemo.netops_dataset.netops_table_alarm`”
Figure 8: Verifying alarms are actually streamed into BigQuery.
Now that our data pipeline is complete, we can move on to visualizing DX NetOps insights in Looker Studio.
Start by configuring a new data source within Looker Studio (Figure 9). Choose BigQuery as the data source type to connect directly to the BigQuery dataset that stores the DX NetOps alarms.
Figure 9: Data source creation in Looker Studio.
Next, create a new report and select the BigQuery data source you just configured. With the data source set up, you can begin enhancing your report by adding various charts and visualizations. This will allow you to effectively showcase network insights, calculated fields, and key performance indicators (KPIs). These views are derived from the raw data that has been ingested into Google Cloud through the streaming data pipeline. Looker Studio’s visualization tools transform your streaming data into dashboards and reports that provide actionable information (Figure 11).
Figure 10: Customization of visualizations in Looker Studio.
Figure 11: Sample report generated by Looker Studio.
This tutorial has guided you through the process of streaming and visualizing DX NetOps insights in BigQuery. By implementing real-time data pipelines like these, network operations teams can leverage the automation and integration capabilities that fuel enhanced insights and efficiency.
Once your data is stored in Google Cloud, there are numerous exciting opportunities to explore, such as leveraging Gemini AI for advanced data analysis—but that will be a topic covered in a future blog post.
To learn more about how DX NetOps by Broadcom collects and analyzes network data at scale, visit our Network Observability microsite.