| Additional contributor: Tomas Kliner |
|
Key Takeaways
|
|
Introduction
DX App Synthetic Monitor (DX ASM) performs synthetic checks from an external perspective to replicate real-user experience. Using the DX ASM global network of more than 90 monitoring stations, customers can test a website or application on a 24/7 basis, with no disruption to production systems. The solution also provides the option to create on-premises monitoring stations (OPMS) within a data center to monitor web applications and APIs inside the firewall.
The newly released Synthetic Operator for DX ASM simplifies deploying and maintaining an on-premises instance of a synthetic monitoring station in a private data center. This first prototype of Synthetic Operator for Kubernetes uses a classic Kubernetes manifest to show the deployment of an on-premises synthetic monitoring station.
Note: The prototype stage for Synthetic Operator only applies to the installer/deployment, as the OPMS code itself is generally available.
Deploy and manage Synthetic Agents with Synthetic Operator
Synthetic Operator is based on the Kubernetes Operator design, which is one of the most commonly accepted patterns for users to automate the tasks required to operate applications on Kubernetes. Synthetic Operator provides some features that specifically help with managing Synthetic Agents and the whole Synthetic ecosystem. This is especially valuable for teams managing large environments that may require custom configurations of a large number of agents.
Many teams use a combination of methods, such as HELM, ConfigMaps, and Terraform, to manage an application’s configuration in the Kubernetes cluster. Synthetic Operator supports CustomResourceDefinitions (CRD) so teams can manage and maintain the synthetic base ecosystem with a single API mechanism. Teams can also manage and maintain synthetic agents as separate units with a separate configuration. This provides powerful options for customizing the local installation to address specific requirements, from the types of agents to run to the sizing of the number of instances for each agent.
To minimize configuration work, Synthetic Operator provides teams with a default configuration they can use for a synthetic base ecosystem and for the agents. So by default, you can deploy "only" the operator, and that is all. But as noted above, you can modify the configuration for each agent based on specific requirements.
Synthetic Operator design
This prototype of Synthetic Operator is designed as a namespace scope (see diagram below). The advantage of this design is that you can deploy the monitoring station in several namespaces. In addition, each namespace can run as a separate and independent cluster with a dedicated node pool for synthetic monitoring. This prevents Synthetic Agents from being limited by other workloads that may run on the same nodes. Synthetic Operator will automatically distribute synthetic workloads through multiple nodes. Each node can then run in different availability zones. This approach enables teams to run Synthetic Agents in high availability (HA) mode.
Because Synthetic Operator can operate in a dedicated namespace and be separated from the rest of the cluster resource, configuration burden is reduced and security is improved.
%20-%20Introducing%20Synthetic%20Operator%20for%20Kubernetes.Figure%201.png?width=323&height=208&name=ESD_FY24_Academy-Blog.DX%20App%20Synthetic%20Monitor%20(ASM)%20-%20Introducing%20Synthetic%20Operator%20for%20Kubernetes.Figure%201.png)
The above diagram shows the logical deployment for HA operations.
A simple example for a non-HA deployment follows. This is a prototype, and will be easier when officially released.
Getting started: How to deploy Synthetic Operator for DX ASM
1. Create new namespace for Synthetic Operator for DX ASM
kubectl create namespace asm-operator-system
Switch to the created namespace:
kubectl config set-context --current--namespace=asm-operator-system
NOTE: After having finished installing the OPMS, remember to set it back to the "defaults" namespace.
2. Apply all role-based access control (RBAC) files from the folder rbac and confirm these have been properly applied
~$ kubectl apply -f ./rbac~$ kubectl get rolesNAME CREATED ATassetcleaner-editor-role 2024-06-11T12:59:41Zassetcleaner-viewer-role 2024-06-11T12:59:41Zcheckpointservice-editor-role 2024-06-11T12:59:41Zcheckpointservice-viewer-role 2024-06-11T12:59:41Zleader-election-role 2024-06-11T12:59:41Zmanager-role 2024-06-11T12:59:41Z
About RBAC authorization
RBAC is a method of regulating access to computer or network resources based on the roles of individual users within your organization.
By default, Kubernetes contains two kinds of roles: Role and ClusterRole. We will focus only on the Role in this article because Synthetic Operator has namespace scope in the current version and Role always sets permissions within a particular namespace. When you create a Role, you must specify the namespace it belongs in.
3. Apply all DX ASM Custom Resource Definitions (CRDs) from the folder crd/bases
~$ kubectl apply -f ./crd/bases~$ k get crdNAME CREATED ATaagents.checkpoint.asm 2024-06-11T13:00:11Zassetcleaners.checkpoint.asm 2024-06-11T13:00:11Ztunnelids.checkpoint.asm 2024-06-11T13:00:11Z... (rest remove - these 3 are important)
When you create a new CRD, the Kubernetes API Server creates a new RESTful resource path for each version you specify. For details, refer to the official Kubernetes documentation for CRD.
4. Create Persistent Volumes (PV) with NFS (because we need to have mode write many to many).
By default, DX ASM components run with the following user/group ownerships:
DefaulFSGroup = 1004DefaultUser = 1010DefaultGroup = 1010$ sudo ls -ld /nfs/k8s/share/opmsdrwxrwx--- 723 1010 1004 28672 Jun 13 14:26 /nfs/k8s/share/opms
Make sure the Persistent directory ownership is writable by the default user UID 1010 and group 1004! Currently these are static (mandatory). We plan to make this a configurable option.
Example PV configuration:
apiVersion: v1kind: PersistentVolumemetadata:name: asm-nfs-server-operatorlabels: type: local app: nfsspec:capacity: storage: 50GiaccessModes: - ReadWriteManynfs: # for example: "/mnt/zpool/asm/nfs/" path: <path_to_nfs_exported_folder> server: <ip_of_the_nfs_server>volumeMode: Filesystem persistentVolumeReclaimPolicy: Retain
Note that persistent volumes definition can be used for on-premises Kubernetes deployments. Some environments will force their own PV.
Create PVC for the assets.
The storage-class can be left empty for on-premises installations. On GKE, you can choose the standard storage class (disk). No fast disks required!
Example PVCapiVersion: v1kind: PersistentVolumeClaimmetadata:name: "asm-nfs-server-pvc"annotations: volume.beta.kubernetes.io/storage-class: ""spec:accessModes: - ReadWriteManyresources: requests: storage: 50GivolumeMode: Filesystem volumeName: "asm-nfs-server-operator"
5. Install Synthetic Operator with this specific version
registry.asm.saas.broadcom.com/asm-operator:0.0.1-rc8
~$ kubectl apply -f ./manager/manager.yamlasm-operator-controller-manager-5fd55c5f4c-2zc4p 1/1 Running 6 (27d ago) 126dasm-operator-controller-manager-5fd55c5f4c-vb4mm 1/1 Running 8 (27d ago) 127d
6. Add redis
Note: Redis is source-available, in-memory storage, used as a distributed, in-memory key-value database.
kubectl apply -f./samples/checkpoint_v1alpha1_checkpointservice_redis.yaml
7. Add smartapi
Note: SmartApi is an internal DX ASM component and also entrypoint to the synthetics ecosystem.
kubectl apply -f ./samples/checkpoint_v1alpha1_checkpointservice_smartapi.yaml
8. Add resultbroker
Note: Resultbroker is an internal DX ASM component which is responsible for reporting results of the checks back to the backend.
kubectl apply -f ./samples/checkpoint_v1alpha1_checkpointservice_resultbroker.yaml
9. Add httpbroker
Note: Httpbroker is an internal DX ASM component that is responsible for translating the monitor definition for other services.
kubectl apply -f ./samples/checkpoint_v1alpha1_checkpointservice_httpbroker.yaml
10. Add tunnelID configuration for optunnel service
Check file: samples/checkpoint_v1alpha1_tunnelid.yaml
TunnelID CRDapiVersion: checkpoint.asm/v1alpha1kind: TunnelIDmetadata:labels: app.kubernetes.io/name: tunnelid app.kubernetes.io/instance: tunnelid-sample app.kubernetes.io/part-of: asm-operator app.kubernetes.io/managed-by: kustomize app.kubernetes.io/created-by: asm-operatorname: tunnelid-samplespec:tunnelID: <replace_with_your_tunnelID_from_ASM_UI>
And load that configuration with:
kubectl apply -f samples/checkpoint_v1alpha1_tunnelid.yaml
Finally, add and activate/load the optunnel itself.
The optunnel connects the OPMS to the DX ASM backend in SaaS. Without this tunnel, no communication with the DX ASM SaaS backend is possible and this endpoint will not show up in the Monitor list. This is critical to ensure that the metrics data is displayed in the frontend.
kubectl apply -f ./samples/checkpoint_v1alpha1_checkpointservice_optunnel.yaml
11. Add agents (jmeter4, cbot, fpm)
kubectl apply -f ./samples/checkpoint_v1alpha1_agent.yaml -f ./samples/checkpoint_v1alpha1_agent_cbot.yaml -f ./samples/checkpoint_v1alpha1_agent_jmeter4.yaml -f ./samples/checkpoint_v1alpha1_agent_php.yaml
12. Add asset cleaner
kubectl apply -f ./samples/checkpoint_v1alpha1_assetcleaner.yaml
Verifying the deployment
Verify the pods themselves, readiness
$ k get pods -n asm-operator-systemNAME READY STATUS RESTARTS AGEasm-cbot-854777d88d-w4zft 2/2 Running 0 8dasm-fpm-7b7bbc9666-68ljn 1/1 Running 8 (9h ago) 8dasm-httpbroker-5cfd8c7c4b-dftvz 1/1 Running 3 (8d ago) 8dasm-jmeter4-8f88686c9-cgwzm 1/1 Running 0 8dasm-optunnel-55fbc76444-mvrx6 1/1 Running 0 8dasm-php-7b487698b9-nrrbf 2/2 Running 0 8dasm-redis-7d559bd49f-gmf2k 1/1 Running 0 8dasm-resultbroker-78fff8c5f-mx7b7 1/1 Running 1 (8d ago) 8dasm-smartapi-5645d99586-cnnkb 2/2 Running 0 8dassetcleaner-sample-28646700-svpt8 0/1 Completed 0 31mcontroller-manager-54c454d8bd-8sb6l 1/1 Running 0 8dcontroller-manager-54c454d8bd-mmwlb 1/1 Running 0 8d
The event should also not show any errors:
$ k get events -n asm-operator-systemLAST SEEN TYPE REASON OBJECT MESSAGE33m Normal Scheduled pod/assetcleaner-sample-28646700-svpt8 Successfully assigned asm-operator-system/assetcleaner-sample-28646700-svpt8 to k8s-jm-node233m Normal Pulled pod/assetcleaner-sample-28646700-svpt8 Container image "registry.asm.saas.broadcom.com/asset-cleaner:0.0.1" already present on machine33m Normal Created pod/assetcleaner-sample-28646700-svpt8 Created container assetcleaner33m Normal Started pod/assetcleaner-sample-28646700-svpt8 Started container assetcleaner33m Normal SuccessfulCreate job/assetcleaner-sample-28646700 Created pod: assetcleaner-sample-28646700-svpt833m Normal Completed job/assetcleaner-sample-28646700 Job completed33m Normal SuccessfulCreate cronjob/assetcleaner-sample Created job assetcleaner-sample-2864670033m Normal SawCompletedJob cronjob/assetcleaner-sample Saw completed job: assetcleaner-sample-28646700, status: Complete33m Normal SuccessfulDelete cronjob/assetcleaner-sample Deleted job assetcleaner-sample-28646640
The last step is to confirm that the monitoring station actually appears in the DX ASM UI. In the "On-Premise" -> Stations tab on the DX ASM UI, you should now see the new on-premise monitoring stations (OPMS).
For more information, refer to the official OPMS documentation: On-Premise Monitoring Stations (OPMS).
%20-%20Introducing%20Synthetic%20Operator%20for%20Kubernetes.Figure%202.png?width=336&height=256&name=ESD_FY24_Academy-Blog.DX%20App%20Synthetic%20Monitor%20(ASM)%20-%20Introducing%20Synthetic%20Operator%20for%20Kubernetes.Figure%202.png)
On the DX Platform, when integrated, you will be able to see all metrics in the metric tree show up under "Synthetic":
%20-%20Introducing%20Synthetic%20Operator%20for%20Kubernetes.Figure%203.png?width=877&height=427&name=ESD_FY24_Academy-Blog.DX%20App%20Synthetic%20Monitor%20(ASM)%20-%20Introducing%20Synthetic%20Operator%20for%20Kubernetes.Figure%203.png)
To summarize
In this blog, we showed the prototype deployment of a DX ASM OPMS (on-premises monitoring station) in Kubernetes. The idea in writing this blog was to provide some background information on which components are deployed, what they do and why they are needed.
Expect the final product to be far easier to deploy. If you need to deploy the DX ASM OPMS on Kubernetes today, reach out to your Broadcom Team so we can guide you through the entire journey.
Jörg Mertin
Jörg Mertin, a Master Solution Engineer on the AIOps and Observability team, is a self-learner and technology enthusiast. A testament to this is his early adopter work to learn and evangelize Linux in the early 1990s. Whether addressing coordinating monitoring approaches for full-fledged cloud deployments or a...
Other resources you might be interested in
This Halloween, the Scariest Monsters Are in Your Network
See how network observability can help you identify and tame the zombies, vampires, and werewolves lurking in your network infrastructure.
Your Root Cause Analysis is Flawed by Design
Discover the critical flaw in your troubleshooting approaches. Employ network observability to extend your visibility across the entire service delivery path.
Whose Fault Is It When the Cloud Fails? Does It Matter?
In today's interconnected environments, it is vital to gain visibility into networks you don't own, including internet and cloud provider infrastructures.
The Future of Network Configuration Management is Unified, Not Uncertain
Read this post and discover how Broadcom is breathing new life into the trusted Voyence NCM, making it a core part of its unified observability platform.
Rally Office Hours: October 9, 2025
Discover Rally's new AI-powered Team Health Widget for flow metrics and drill-downs on feature charts. Plus, get updates on WIP limits and future enhancements.
AAI - Navigating the Interface and Refining Data Views
This course introduces you to AAI’s interface and shows you how to navigate efficiently, work with tables, and refine large datasets using search and filter tools.
Rally Office Hours: October 16, 2025
Rally's new AI-driven feature automates artifact breakdown - transforming features into stories or stories into tasks - saving time and ensuring consistency.
What’s New in Network Observability for Fall 2025
Discover how the Fall 2025 release of Network Observability by Broadcom introduces powerful new capabilities, elevating your insights and automation.
Modernizing Monitoring in a Converged IT-OT Landscape
The energy sector is shifting, driven by rapid grid modernization and the convergence of IT and OT networks. Traditional monitoring tools fall short.