Organizations rely on networks for a wide array of critical business functions, including those that support employee productivity and customer-facing services. In today's organizations, however, these networks are often not owned or directly managed by the enterprise’s internal IT and network operations teams. With the shift to hybrid work and cloud solutions, network operations teams lose the ability to validate the network performance and the user experience across third-party infrastructure.

Traditional passive monitoring of real user traffic provides the ability to react when users are impacted by poor network quality, but relies on user activity for performance insight and may not have enough data for network operations teams to identify issues before users are impacted. Monitoring internally managed infrastructure allows for quick identification and remediation of internal network issues, but does not provide the network operations team the data required to identify which of the external networks is underperforming.

To address this problem, network operations teams need active monitoring for continuous performance validation, which gives them the proactive insight needed to understand quality on any network, for any user.

This article presents a learning path for network operations teams to learn how to assure network quality with end-to-end path analytics, as described in the Assure Network Quality white paper.

Network Observability by Broadcom is a solution that brings together active and passive network monitoring approaches. The solution combines the automated discovery capabilities for the internally managed network provided by DX NetOps with the active external network monitoring provided by AppNeta’s TruPath™ technology.

​​The Broadcom approach to provide network operations teams with the ability to assure network quality across third-party infrastructure involves:

• AppNeta Monitoring Points are strategically placed on the network according to the organization’s specific network architecture and business needs to actively monitor the end-to-end network performance of a business-critical network or application service.
• Continuous active network monitoring to provide visibility into layer 3 network performance and traffic routes between end users and the business-critical network and application services they rely on.

AppNeta Monitoring Points

Monitoring Points are the hardware, software, or virtual installations that monitor network and application performance as part of an AppNeta deployment. Monitoring Points installed at key locations in the network monitor important targets from the perspective of end users or business services at those locations, then send data to the AppNeta application for analysis, alerting, and reporting. To learn about the features supported by each Monitoring Point model, refer to the AppNeta documentation article: Monitoring Point Feature Comparison.

Active network monitoring for Layer 3 network performance visibility

Active network monitoring with AppNeta is accomplished through the TruPath™ packet train dispersion technology. Monitoring Points use the TruPath technology to send and receive many varied short sequences of packets, called packet trains. Packet trains are transmitted to defined end hosts or targets, which can be any endpoint that can respond to an ICMP-based ping or can send back a TCP or UDP packet.

Using this technology, TruPath can build up a complete set of network statistics very quickly—in many cases, in just tens of seconds. TruPath uses special patterns designed to detect if instrumentation packets are interfering with each other. If that happens, it takes more varied samples over a longer time scale to ensure that the resulting statistics are clean.

By sending multiple sets of distinct packet sequences, TruPath can analyze a wide range of traffic conditions that a user on a network path might experience. By probing the path repeatedly with the packet sequences, TruPath collects a statistically significant collection of responses for each type. TruPath will detect when samples are captured during times of rapidly changing conditions and adjust its measurement patterns accordingly.

Unlike packet flooding technologies available on the market, this approach delivers high accuracy without requiring an intrusively high instrumentation load on the network path. With the technology’s low-overhead approach, network operations teams can continuously run TruPath on production networks for end-to-end visibility, without impacting the network.

To learn more about the TruPath technology, refer to the AppNeta documentation article: Understanding Network Monitoring.

Configuring the solution to assure network quality involves the following procedures:

• Create an AppNeta monitoring plan.
• Deploy AppNeta Monitoring Points.
• Set up active network monitoring with monitoring policies in AppNeta.
• Customize alert profiles in AppNeta.
• Integrate AppNeta with DX NetOps for alarm management and incident detection.
• Incorporate AppNeta monitoring into network operations procedures.

Create an AppNeta monitoring plan

Network monitoring in AppNeta is configured using monitoring policies. Policies are most effective and easiest to manage with clear, simple rules to define where to monitor from. To ensure clear, simple rules, we recommend first creating a plan for what you want to monitor and where you want to monitor it from. In the plan, put each Monitoring Point that will monitor a target into logical groups and determine how the groups of Monitoring Points will be selected in the policy. Monitoring Point selection in a policy can be based on Monitoring Point tags, such as Monitoring Point type, geographic location, or any custom tags assigned to Monitoring Points, and on network rules, such as the subnet.

When creating an active monitoring plan, consider the network architecture, including critical points of ingress and egress, common traversal points, and cloud services. Specifically, consider:

• Locations and networks that user traffic and business-critical services traverse, including common traversal points, such as firewalls, secure web gateways, DMZ placement, and so on.
• Whether the organization is using cloud services, and, if so, which specific regions traffic traverses.
• Any services provided to external customers where outside-in monitoring is required. 
• The level of service that business users expect, including circuit speeds, and how active testing will validate whether those expectations are being met.
• User personas and how they interact with network infrastructure, such as:
  • Connectivity, including SD-WAN, VPN, proxy, and LAN/Wi-Fi.
  • Workstation types, including laptops or desktops, virtual desktop infrastructures (VDI), and thin clients.
  • Top applications used, whether business-critical or otherwise.
  • End user population size per location.
• The different paths data may take as it moves across the network, such as from an office to a data center over one of two ISP connections.
• Any business function context for monitored targets and locations targets are monitored from.

The following diagram represents an example AppNeta monitoring plan to assure network quality from Monitoring Points deployed at branch offices and on end-user workstations to two regional data centers, a virtual private cloud (VPC) environment, and three SaaS applications. In this example, users at branch offices access services in one of two regional data centers, and each office relies on two ISPs for connectivity. Additionally, the business function context identifies workstations by department to determine the relevant SaaS applications to monitor from particular workstations.

Diagram illustrating the example plan for monitoring network quality.

Deploy AppNeta Monitoring Points

For each location or workstation that should monitor network quality, plan to deploy an appropriate Monitoring Point. Select a Monitoring Point model that supports the type of monitoring you want to configure from each location. To learn about the differences between each Monitoring Point model, refer to the AppNeta documentation article: Monitoring Point Feature Comparison.

Typically, native Monitoring Points (NMP) are recommended for user workstations, r1000 physical Monitoring Points are recommended for data centers, and physical or virtual Monitoring Points are recommended for offices. Monitoring Points are also available for deployment to customer premise equipment, such as Cisco Catalyst 9300/9400 switches. For cloud environments you control, plan to deploy a container-based Monitoring Point (CMP) within the same cloud environment as your web application infrastructure to act as a monitoring target.

For outside-in monitoring, Global Monitoring Points (GMP) deployed and managed by Broadcom can be used to monitor and validate performance from outside your network. To learn more about Global Monitoring Points, refer to the documentation article: Global Monitoring Points (GMP).

To learn more about how to deploy Monitoring Points, refer to the following AppNeta documentation article and select the model from the Set up a Monitoring Point section: Getting Started.

Tag Monitoring Points

Assign custom tags to Monitoring Points with relevant business function context. Tags can be used in monitoring policies to automatically start monitoring targets from Monitoring Points with the selected tags. Tags are also inherited by network paths, making it easier to search and filter for monitoring results.

The following video introduces ​​best practices for tagging Monitoring Points, monitoring policies, and paths in AppNeta.

The following video demonstrates how to configure and manage custom Monitoring Point tags in AppNeta.

Set up active network monitoring with monitoring policies in AppNeta

Use monitoring policies in AppNeta to carry out the plan for which applications and networks to monitor, and from where. Dynamic policy rules enable you to automatically set up monitoring from new Monitoring Points that match the policy rules.

In the example monitoring plan, the network administrator would create a monitoring policy that targets the Monitoring Point deployed in the US data center and customize the monitoring preferences for dual-ended monitoring. Monitoring Point rules in the policy would automatically include physical Monitoring Points tagged with the custom tag Site Type: Branch Office and the location tag Region: Americas. Network rules in the policy could be used to monitor the target across each ISP connection.

The network administrator would then create additional monitoring policies for each target in the plan.

Note: If web performance monitoring will be conducted in AppNeta, a single-ended network monitoring policy is not required to monitor the network connection to the application. When web monitoring is configured, single-ended network monitoring is automatically configured.

The following video demonstrates how to configure a monitoring policy in AppNeta to monitor network performance.

To learn more about how to set up active network monitoring in AppNeta, refer to the following resources:

• AppNeta: Configure Network Monitoring (Online Course)
• Set Up Delivery (AppNeta Documentation)

Customize alert profiles in AppNeta to align with network SLAs

In AppNeta, an alert profile is the set of conditions and customizable thresholds that define the limits of acceptable performance. When AppNeta detects a violation of an alert profile condition, the system logs an event.

If configured by an admin, the event can trigger a notification by email, SNMP trap, or webhook.

For metrics other than connectivity loss, AppNeta may also automatically trigger a diagnostic test.

In most scenarios, the default alert profile is sufficient. You may need to customize an alert profile if your organization has existing service-level agreements (SLAs) or specific performance requirements for the monitored services.

The default network path alert profile includes alert conditions to track violation events when connectivity to the target is lost for one minute, when data loss is greater than 2% for two minutes, and when data jitter is greater than 100 ms for two minutes.

To learn more about how to create custom network path alert profiles in AppNeta, refer to the following resources:

• AppNeta: Configure Network Monitoring (Online Course, refer to the Performance Metrics and Alerts lesson)
• Delivery Alerts (AppNeta Documentation)

Integrate AppNeta with DX NetOps for alarm management and incident detection

DX NetOps provides a unified portal to monitor, analyze, visualize, and raise alarms based on performance data collected from multiple vendors and technologies.

Integrate AppNeta with DX NetOps to pull network performance data from AppNeta to DX NetOps for baselining, alarm generation, and incident detection alongside performance monitoring of managed networks. To learn how to integrate AppNeta and DX NetOps, refer to the online course: DX NetOps 23.3.x: Integrate AppNeta 200.

Note: DX NetOps consists of a set of software components that are deployed on-premises and then integrated to power the different DX NetOps capabilities. Before integrating AppNeta with DX NetOps for alarm management and incident detection, the DX NetOps environment must include the following integrated software components: DX NetOps Performance Management for performance monitoring, baselining, dashboards, and the DX NetOps Portal UI. DX NetOps Virtual Network Assurance to collect performance data from AppNeta. DX NetOps Spectrum for fault monitoring, alarm management, and correlated incident detection for monitored targets. To learn more about DX NetOps architecture, installation, configuration, and software component integration, refer to the learning path: DX NetOps Installation and Configuration.

Configure threshold profiles in DX NetOps

Baselines and standard deviations for AppNeta performance metrics calculated by DX NetOps enable operators to see when performance values are changing in statistically meaningful ways. The solution also presents baseline data that helps teams to assess present performance and estimate future performance. To learn how DX NetOps calculates baseline averages and standard deviations, refer to the documentation article: Baseline Calculations.

Create threshold profiles in DX NetOps to raise or clear threshold violation events, for example, if a network performance metric deviates from baseline performance. To learn how to configure threshold profiles, refer to the online course: DX NetOps 23.3.x: Configure Performance Thresholds and Notifications 200.

Configure correlated incident detection for AppNeta monitored targets

Once AppNeta is integrated with DX NetOps, the solution can optionally correlate network performance and connectivity events across locations monitoring the same target and raise a single alarm for the correlated incident. 

Example of a critical severity alarm raised in DX NetOps when 100% of network paths lost connection to a monitored target.

To avoid creating unexpected alarms, monitored target alarm thresholds are not configured by default. Configure correlated incident detection for each monitored target in the DX NetOps Spectrum OneClick web application. For each monitored target, set thresholds for the percentage of network paths monitoring the target that must be experiencing an event for alarms of different severity levels to be created. 

Note: For the purpose of event correlation, an event is any event for a network path, which could include threshold profile violation events detected by DX NetOps and events created from SNMP traps received from AppNeta. Events on web paths are not currently considered for correlation.

A recommended approach to defining the severity level for a monitored target is:

• Critical: A loss of service has occurred. Immediate action is required.
• Major: A loss of service has occurred or is impending. Action is required within a short period of time.
• Minor: A situation has occurred that does not require immediate action. This severity is also used for alarms created to convey information only.

Example of a monitored target in DX NetOps Spectrum OneClick. In this example, the Monitored Target Alarm Threshold Configuration settings are configured to raise a minor alarm when events occur on 40% or more of network paths monitoring the target, a major alarm when events occur on 50% or more of network paths monitoring the target, and a critical alarm when events occur on 60% or more of network paths monitoring the target.

To configure correlated incident detection for AppNeta events in DX NetOps:

1. Log into the DX NetOps Spectrum OneClick web application.
   1. In the Explorer tab of the Navigation panel, navigate to the Monitored Targets container.
   2. To locate Monitored Targets in the Explorer tab of the Navigation panel, navigate to the landscape that contains AppNeta data, then SDN Manager, then the VNA Gateway, then the AppNeta Domain Container, then Technologies, then the AppNeta container, then expand the Monitored Targets container.
2. Each item nested within the Monitored Targets container is a target monitored by one or more web or network paths in AppNeta.
3. For each monitored target:
   1. In the Explorer tab of the Navigation panel, select a monitored target.
   2. In the Contents panel, select the Information tab.
   3. On the Information tab, expand Monitored Target Alarm Threshold Configuration.
   4. For each alarm severity level, click the set button in the Set Threshold column, then enter a value above which an alarm is created. Press Enter to save the value.
      1. The Set Threshold value represents the percentage of paths that must experience an event for an alarm to be created.
   5. For each alarm severity level, click the set button in the Reset Threshold column, then enter a value below which an existing alarm condition is cleared. Press Enter to save the value.
      1. The Reset Threshold value represents the percentage of paths below which an existing threshold alarm is automatically cleared. 

Steps to configure monitored target alarm thresholds in DX NetOps Spectrum OneClick include: a) select a monitored target in the Explorer tab, b) select the Information tab in the Contents panel, c) expand Monitored Target Alarm Threshold Configuration, d) set the thresholds for minor, major, and critical alarms, and e) set the reset threshold for minor, major, and critical alarms.

To learn more about navigating the DX NetOps Spectrum OneClick web application, refer to the online course: DX NetOps 23.3.x: Navigating OneClick for Spectrum 200.

Incorporate AppNeta monitoring into network operations procedures

Prepare investigation runbooks

Consider how the network operations team should handle performance events detected by AppNeta, and where they would expect to find runbooks. For runbooks with general steps to investigate violation events with AppNeta, refer to the following resources:

• Investigating Violation Events (AppNeta Documentation)
• AppNeta: Configure Network Monitoring (Online Course; refer to the Course Introduction lesson to download a customizable runbook template)

Periodically review active monitoring deployment

Plan a regular cadence to review the monitoring deployment, tune alert thresholds and notifications, and assess any remaining or new gaps in monitoring.

Consider whether the network operations team is receiving too many alerts (or too few). Broadcom recommends integrating AppNeta with DX NetOps to reduce alarm noise with advanced out-of-the-box analysis, including deviation-from-normal baselines and correlated incident detection for common monitored targets.

If the operations team receives too many alerts generated directly from AppNeta, review and fine-tune alert profile thresholds in AppNeta. For more information on how to fine-tune AppNeta alerts and notifications, refer to the following online course: AppNeta: Operationalize Alerts and Notifications.

Similarly, if the network operations team is not alerted to degraded performance before users report issues, review and fine-tune thresholds and dynamic baseline parameters as necessary. 

The scope of responsibility for the network operations team has expanded significantly. Network operations teams are often held accountable when there is a unified communication service glitch or an application performance issue. These teams now find themselves in a defensive role, where the assumption of guilt prevails until the network is proven innocent. This explains why a metric called mean time to innocence (MTTI) has gained increasing prominence alongside mean time to resolution (MTTR). MTTI is how long it takes for the network operations team to prove a degradation is not caused by the part of the network they own. Without effective solutions to assist them, operations teams face challenges in reducing MTTI and proving their innocence when issues arise outside of the networks they control.

AppNeta monitoring helps identify and isolate issues within and between the various network environments transited by production application traffic. When issues occur, AppNeta helps network operations teams quickly identify where the issue was introduced—reducing the uncertainty of who is responsible for resolution—and gather supporting evidence for external service providers.

Identify the error domain to reduce MTTI

The goal when investigating an issue detected by AppNeta is to quickly narrow down the error domain, the term we'll use for the network segment responsible for degraded performance experienced by test packets. For example, when monitoring performance between end-user workstations and cloud-hosted applications, the error domain could be the workstation, the user’s wireless environment, the last-mile Internet service provider (ISP), the cloud edge, cloud backbone, the application environment, or the application itself.

AppNeta route visualization highlighting the network hops that are part of the user domain, ISP or transit domain, and cloud domain.

A typical approach to investigating degraded performance starts with understanding how widespread an issue is. Sometimes this is all that is needed to isolate the error domain.

1. How widespread is the problem?
   Review which paths are experiencing similar issues at the same time for a quick indication of where the issue might be. For example, if all paths targeting the same application are experiencing the same issue, the problem is likely with the target application, its network infrastructure, or some network infrastructure shared by all paths.
2. What are the symptoms for impacted paths?
   Review the metric timeline charts for patterns and anomalies leading up to and during the period of degraded performance. Regular patterns can indicate a performance impact due to a scheduled activity, for example, scheduled data backups occurring at the same time every day at the source location.
3. What infrastructure do the impacted paths have in common?
   Compare routes for multiple paths and look for commonalities across impacted paths. Check whether routes changed over time before and during the period of degraded performance. Review any diagnostic tests that ran while performance was degraded. Check for anomalies or unexpected performance changes along the route. To identify the error domain, look for both the presence of an anomalous result (for example, packet loss or a drop in MOS) at a hop along the route and its persistence through each subsequent hop to the target.
4. Next Steps:
   ◦ If the issue is introduced at a WAN hop within a provider network, correlate results with additional diagnostics before contacting the provider.
   ◦ If an issue is suspected to be introduced at a WAN egress router or ISP connection point, investigate the possibility of overutilization at the source location. If available, review Usage results collected by the relevant Monitoring Point for further insight.
   ◦ If an issue is suspected to be introduced at a LAN hop, investigate the device itself, the preceding device, and the infrastructure in between.
   ◦ If an issue is suspected to be introduced by an individual’s workstation or wireless environment, review the host and wireless metrics for supporting evidence.
   ◦ To determine if a network event impacted the user experience of a web application, review layer 7 web performance metrics for the related web paths, if available.

To learn more about how to investigate performance issues with AppNeta, refer to the following resources:

• Intro to AppNeta Results (Online Course)
• Investigating Violation Events (AppNeta Documentation)
• Network Path Routes (AppNeta Documentation)
• Diagnostics (AppNeta Documentation)

Understand the metrics

Active network monitoring generates the following performance metrics displayed as charts in the Delivery area of AppNeta. Network performance metrics can be classified as either primary metrics, which are directly measured and used to detect network issues, or derived indicators, which are either derived or inferred and used to assist with diagnosis.

Primary network performance metrics:

• Connectivity indicates whether the Monitoring Point can connect to the target. Connection loss occurs with 100% packet loss.
• Packet loss is simply a measure of the number of packets that did not make it to their intended destination. AppNeta provides metrics for Data Loss, based on packet trains emulating data traffic, and Voice Loss, based on packet trains emulating voice traffic.
• Round-trip time (RTT) ​​is the time it takes for a packet to go from the source Monitoring Point to the target and back.
• Jitter is a measure of variation in latency. AppNeta provides metrics for Data Jitter, based on packet trains emulating data traffic, and Voice Jitter, based on packet trains emulating voice traffic.

Derived network performance indicators:

• Latency is the time it takes for a packet to go from a source to a target. AppNeta calculates latency as one-half of the RTT of the fastest packet in a packet train.
• Mean Opinion Score (MOS) is an estimate of the rating a typical user would give to the sound quality of a call. It is expressed on a scale of 1 to 5, where 5 is perfect. It is a function of loss, latency, and jitter. It also varies with voice codec and call load.
• Capacity measures the end-to-end transmission rate observed for small bursts of traffic, between a source Monitoring Point and a given target. Total Capacity is the peak transmission rate as calculated by the TruPath technology. Available Capacity is the portion of total capacity that is available for use.

Additional wireless metrics, such as Signal Quality, Link Speed, RSSI, and host metrics, such as CPU, Memory, and Top Processes, are available for supported Monitoring Points.

To learn more about network performance metrics generated by AppNeta, refer to the documentation article: Network Path Performance.

Broadcom can address the challenge of assuring network quality by providing active monitoring of external networks from AppNeta Monitoring Points placed strategically on the network to restore visibility. With the Broadcom solution, network operations teams gain the visibility required to validate network performance meets expectations, and to identify the error domain when performance issues arise.​

Schedule a demo to see exactly how Network Observability by Broadcom can help you assure network quality. Or, for more information on how to implement Network Observability by Broadcom for other use cases, explore other learning paths.