<img height="1" width="1" style="display:none;" alt="" src="https://px.ads.linkedin.com/collect/?pid=1110556&amp;fmt=gif">
November 27, 2021

Employing RESTmon to Ingest Zabbix Data: Part 1

by: Saurabh Sharma

Introduction

DX Operational Intelligence can consume both structured and unstructured data, including topology data, metrics, and alarms. Through RESTMon, teams can ingest data from a range of third-party data sources.

RESTmon offers these key capabilities:

  • Ingest data like metrics and alarms
  • Poll data from an external application via the REST API
  • Stream data from third-party applications
  • Ingest data in XML and JSON
  • Transfer over HTTPS

In this two-part skill builder, we’ll outline how to set up RESTmon and employ it to ingest data from Zabbix. The first part offers step-by-step guidance for deploying a RESTmon instance and preparing it to ingest Zabbix data. The second part will detail how to configure a Zabbix schema for consumption.

For more information on RESTmon please refer to the official documentation. In addition, you can read this blog post: “Unify Visibility Across Your Monitoring Tools with DX Operational Intelligence.”

As you review this document, here are a few things to note:

  • In some places you will notice a “k” which is an alias I use for the command “kubectl.”
  • In cases in which the version for a library or utility is not specified, assume the latest version is required.

Installing RESTmon

In this section, I will offer step-by-step instructions for installing RESTmon. You can also find detailed technical documentation here.

Host Machine: CentOS

I am using a Linux machine to deploy the instance.

> cat /etc/redhat-release

CentOS Linux release 7.9.2009 (Core)

 

 

 

Broadcom Enterprise Software Academy - Employing RESTmon to Ingest Zabbix Data: Part 1

Prerequisites

For the machine that you are deploying on, make sure you have the following apps installed:

  • Docker, version 18 and above
  • Helm, version 3
  • Kubernetes, version 1.20 or later

On my machine I have the following versions:

Docker Version

Client: Docker Engine – Community
Version:           20.10.9
API version:       1.41
Go version:        go1.16.8
Git commit:        c2ea9bc
Built:             Mon Oct  4 16:08:14 2021
OS/Arch:           linux/amd64
Context:           default
Experimental:      true

Helm Version

version.BuildInfo{Version:”v3.2.4”, GitCommit:”0ad800ef43d3b826f31a5ad8dfbb4fe05d143688”, GitTreeState:”clean”, GoVersion:”go1.13.12”}

Kubernetes Version

Client Version: version.Info{Major:”1”, Minor:”22”, GitVersion:”v1.22.2”, GitCommit:”8b5a19147530eaac9476b0ab82980b4088bbc1b2”, GitTreeState:”clean”,
BuildDate:”2021-09-15T21:38:50Z”, GoVersion:”go1.16.8”, Compiler:”gc”, Platform:”linux/amd64”}

Server Version: version.Info{Major:”1”, Minor:”22”, GitVersion:”v1.22.1”, GitCommit:”632ed300f2c34f6d6d15ca4cef3d3c7073412212”, GitTreeState:”clean”,
BuildDate:”2021-08-19T15:39:34Z”, GoVersion:”go1.16.7”, Compiler:”gc”, Platform:”linux/amd64”}

Step 1: Download the Package

Download the package from the support site. At the time of writing, the latest package available is 2.1.3.

Broadcom Enterprise Software Academy - Employing RESTmon to Ingest Zabbix Data: Part 1

I chose the FTP option to download the binary.

Broadcom Enterprise Software Academy - Employing RESTmon to Ingest Zabbix Data: Part 1

Step 1.1 Download the FTP Package

sftp <user>@downloads.broadcom.com

sftp> ls
23135811 
sftp> ls 23135811/

23135811/RESTMon Container 2.1.3.zip

sftp> get 23135811/RESTMon Container 2.1.3.zip

/23135811/RESTMon Container 2.1.3.zip                                                                                                                                                              100%  310MB 635.4KB/s   08:19

sftp> bye

 

This downloads the zip file onto your local system. Once downloaded, you can rename the file as desired (e.g. RESTMON-2.1.3.zip).

Step 1.2 Unzip the Package: RESTMON.zip

root@master>unzip Restmon-2.1.3.zip
Archive:  Restmon-2.1.3.zip
   creating: docker/
  inflating: docker/docker-compose.yaml 
  inflating: docker/README.txt       
  inflating: dx-restmon-2.1.3.tar.gz 
   creating: helm/
  inflating: helm/README.txt         
extracting: helm/restmon-2.1.3.tgz 
  inflating: README.txt

Step 2: Load the Image: dx-restmon-2.1.3.tar.gz

docker load < dx-restmon-2.1.3.tar.gz
e2eb06d8af82: Loading layer [==================================================>]  5.865MB/5.865MB
aabf79748299: Loading layer [==================================================>]  6.668MB/6.668MB
0a1c430491bf: Loading layer [==================================================>]  10.13MB/10.13MB
c42c37cd4d81: Loading layer [==================================================>]   25.8MB/25.8MB
14c5de161296: Loading layer [==================================================>]  14.08MB/14.08MB
30f32f432e48: Loading layer [==================================================>]  263.6MB/263.6MB
883a6d192573: Loading layer [==================================================>]  2.048kB/2.048kB
03828a96ddf5: Loading layer [==================================================>]  15.87kB/15.87kB
843cde0def8f: Loading layer [==================================================>]  198.2MB/198.2MB
Loaded image: dx-restmon:2.1.1

The last statement, “Loaded image: dx-restmon:2.1.1,” confirms the image has been successfully loaded.

Step 2.1: Validate the Image Presence

You can validate the image by issuing the command “docker image ls” and looking for the dx-restmon image.

root@master>docker image ls
REPOSITORY                                   TAG       IMAGE ID       CREATED         SIZE
dx-restmon                                   2.1.1     004b60320570   4 weeks ago     522MB

 

Step 3: Deploy Using Helm

It is important to refer to the README.txt as it outlines the contents and the steps, which we will be following in this skill builder.

Broadcom Enterprise Software Academy - Employing RESTmon to Ingest Zabbix Data: Part 1

As indicated in point 1 in the README.txt, you need token details to connect to DX Operational Intelligence, which can be obtained from the solution.

Step 3.1: Generate token

Navigate to Settings > Connector Parameters.

Broadcom Enterprise Software Academy - Employing RESTmon to Ingest Zabbix Data: Part 1

Broadcom Enterprise Software Academy - Employing RESTmon to Ingest Zabbix Data: Part 1Note the Cohort ID, which is an alphanumeric string.

Broadcom Enterprise Software Academy - Employing RESTmon to Ingest Zabbix Data: Part 1

Click “Generate New Token.”

Broadcom Enterprise Software Academy - Employing RESTmon to Ingest Zabbix Data: Part 1

And copy the token by clicking the Copy button.

Broadcom Enterprise Software Academy - Employing RESTmon to Ingest Zabbix Data: Part 1

Step 3.2: Persistent Volumes

Persistent volumes abstracts how storage is provided and consumed using storage classes. You can read more about storage classes in the Kubernetes documentation. In general, these classes allow you define attributes for the kind of storage available.

Defining the Following Persistent Volume: ssdnfspv

Create a file `pv.yaml` with contents as detailed below.

apiVersion: v1
kind: PersistentVolume
metadata:
   name: ssdnfspv
spec:
  capacity:
   storage: 5Gi
  accessModes:
   - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName:
  nfs:
   path: /var/nfs
   server: server.broadcom.com

The PV can now be created using the following command: kubectl create -f pv.yaml

> kubectl create -f pv.yaml
persistentvolume/ssdnfspv created

You can see the listing by issuing the following command: kubectl get pv

> kubectl get pv
NAME           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                                STORAGECLASS   REASON   AGE
ssdnfspv       5Gi        RWO            Retain           Available   

The persistent volume is now available for claims.


Helm is a Kubernetes package manager.


Before we deploy RESTmon using Helm, let’s look at the possible values we can supply to the chart for customization.


Value File: You can learn more in the official documentation here.


values.yaml: restmon-2.1.3.tgz

The “values.yaml” file allows you to define the values that you can feed to the chart before it is installed.


helm install [NAME] [CHART] [flags]


In the directory that contains the Helm chart, “restmon-2.1.3.tgz” you can use the “helm show all <chart-name>” to get more information.

root@master>helm show all restmon-2.1.3.tgz

Following is the output.

apiVersion: v1
appVersion: "1.0"
description: A Helm chart for restmon-k8s
name: restmon
version: 2.1.11

---

# Default values for restmon.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
restmon:
  id: 1
  storage:
    size: 5Gi
    className: ssd
    nfs_path:
    nfs_server:
  imageName: dx-restmon:2.1.3-29
  service:
    type: LoadBalancer
  resources:
    #  Add resources needed and remove the cpu request and memory section
    requests:
      memory: 3Gi
      cpu: 2
    limits:
      memory: 3Gi
      cpu: 2
  readinessProbe:
    enable: true
    initialDelaySeconds: 60
    periodSeconds: 30
    timeoutSeconds: 30
    successThreshold: 1
    failureThreshold: 5
  livenessProbe:
    initialDelaySeconds: 60
    periodSeconds: 30
    timeoutSeconds: 10
    successThreshold: 1
    failureThreshold: 5
# Enable key value pair for nodeSelector, only when the worker node label is matching with provided value (disktype=ssd)
  nodeSelector:
    #disktype: "ssd"
  settings:
    restmon_protocol: "http"
    restmon_port: ""
    service_port: ""
    log_level: "INFO"
    replace_oi_attributes: "true"
    jarvis_ingestion_hostname: "api.dxi-na1.saas.broadcom.com"
    jarvis_ingestion_port: "443"
    jarvis_ingestion_protocol: "https"
    oi_ingestion_api_host: "apmgw.dxi-na1.saas.broadcom.com"
    oi_ingestion_api_port: "443"
    oi_ingestion_api_protocol: "https"
    oi_ingestion_tenant_token: "<OI/APM Agent Token>"
    tenant_id: "<Tenant Cohort ID>"
    max_number_of_docs: "1000"
    restmon_min_memory_limit: 512M
    restmon_max_memory_limit: 2G
    supportability_agentName: SuperDomain|newoiservices|opendataconnector
    supportability_instanceName: masterdefault
    ssl_key_store_type: PKCS12
    ssl_key_store: restmon.keystore-new
    ssl_key_store_password: restmon
    ssl_key_store_alias: restmonhttpsJune2021
    user_name: admin
    password: password
    readiness_check_interval_sec: "15"
    profile_queue_size_max_limit: "20000"
    profile_queue_size_min_limit: "15000"
    max_allowed_running_profiles: "5"
    # Check interval defaults to 15 seconds
    db_health_check_interval_sec: "15"
    health_statistics_log_level: "ERROR"
    liveness_check_interval_min: "15"
    unready_profile_max_allowed_retries: "10"
    unready_profile_retry_time_sec: "10"
    log_file_max_size: "50MB"
    log_file_max_history: "10"
    show_swagger: "true"
    reload_ootb_schemas: "false"

---

These are the default values. Let’s override some specific values.

NFS Properties

className:
nfs_path: /var/nfs
nfs_server: server.broadcom.net

Image Name

imageName: dx-restmon:2.1.1

Node Selector

nodeSelector:
   kubernetes.io/hostname: node1.broadcom.net

Token

oi_ingestion_tenant_token: "TjXWZeR9nrLAIDDDawa6ezQ"

Tenant ID

tenant_id: "6AAAAAAAAAAAÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀC"

Instance Name

supportability_instanceName: restmom_demo_deployment

DEBUG Logs

health_statistics_log_level: DEBUG

Step 3.3 Deploy the RESTMON Instance

With the “values.yaml” defined, we can now release.

helm install demo-restmon ./restmon-2.1.3.tgz --values ./values.yaml

“helm ls” will allow you see the release listing.

> helm ls

NAME                    NAMESPACE       REVISION        UPDATED                                                     STATUS          CHART                           APP VERSION    
demo-restmon            default         1                  2021-10-13 13:05:51.797951614 +0000 UTC deployed        restmon-2.1.11                  1.0

 

You can also check the pods deployed using the following command: kubectl get pods

> k get all -l app=dx-restmon1 -o wide

NAME                                           READY   STATUS    RESTARTS   AGE   IP               NODE                           NOMINATED NODE   READINESS GATES
pod/dx-restmon-deployment-1-658d967944-j9qlt   1/1     Running   0          60m   172.22.165.202   ibnqa003969.bpc.broadcom.net   <none>           <none>

NAME                   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE   SELECTOR
service/dx-restmon-1   NodePort   10.111.13.151   <none>        8080:32088/TCP   60m   app=dx-restmon1

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES             SELECTOR
deployment.apps/dx-restmon-deployment-1   1/1     1            1           60m   dx-restmon   dx-restmon:2.1.1   app=dx-restmon1

NAME                                                 DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES             SELECTOR
replicaset.apps/dx-restmon-deployment-1-658d967944   1         1         1       60m   dx-restmon   dx-restmon:2.1.1   app=dx-restmon1,pod-template-hash=658d967944

 

Let’s try accessing the swagger UI using the NodePort.

Broadcom Enterprise Software Academy - Employing RESTmon to Ingest Zabbix Data: Part 1

This confirms RESTmon is up and running.

Ingress Controller

Ingress

Ingress is an API object that manages external access to the services in a cluster, typically HTTP. Ingress may provide load balancing, SSL termination, and name-based virtual hosting. (SSL termination is the process of decrypting SSL-encrypted data traffic.)

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the ingress resource.

An ingress may be configured to give services externally reachable URLs, load balance traffic, terminate SSL / TLS, and offer name-based virtual hosting.


One must have an ingress controller to satisfy an ingress.


In our case, I am using HAProxy Ingress controller.

Quick Guide

Deploy the HAProxy ingress controller using Helm.

Access official documentation for the community edition of the HAProxy ingress controller.


helm install kubernetes-ingress haproxytech/kubernetes-ingress

> helm ls

NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                  APP VERSION
kubernetes-ingress      default         1               2021-10-14 06:48:00.671735715 +0000 UTC deployed        kubernetes-ingress-1.17.4    1.7.0 

View the “kubernetes-ingress” pods.

> k get all -owide --show-labels=true -l app.kubernetes.io/name=kubernetes-ingress

Pods

NAME                                      READY   STATUS    RESTARTS      AGE    IP              NODE                            NOMINATED NODE   READINESS GATES   
pod/kubernetes-ingress-64b5698b4b-bhscj   1/1     Running   2 (97m ago)   2d1h   172.16.25.158   lvnbld011256.bpc.broadcom.net   <none>           <none>            
pod/kubernetes-ingress-64b5698b4b-ck6dt   1/1     Running   2 (97m ago)   2d1h   172.16.25.161   lvnbld011256.bpc.broadcom.net   <none>           <none>            

Service

NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)               AGE    SELECTOR               LABELS
service/kubernetes-ingress     NodePort    10.101.156.196   <none>        80:30125/TCP,443:32388/TCP,1024:30313/TCP   2d1h 
app.kubernetes.io/instance=kubernetes-ingress,app.kubernetes.io/name=kubernetes-ingress                   app.kubernetes.io/instance=kubernetes-ingress,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=kubernetes-ingress,app.kubernetes.io/version=1.7.0,helm.sh/chart=kubernetes-ingress-1.17.4
service/kubernetes-ingress-default-backend   ClusterIP   None             <none>        8080/TCP                                    2d1h 
app.kubernetes.io/instance=kubernetes-ingress,app.kubernetes.io/name=kubernetes-ingress-default-backend   app.kubernetes.io/instance=kubernetes-ingress,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=kubernetes-ingress,app.kubernetes.io/version=1.7.0,helm.sh/chart=kubernetes-ingress-1.17.4

Deployment

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS                           IMAGES                                 SELECTOR                                                                                                  LABELS
deployment.apps/kubernetes-ingress                   2/2     2            2           2d1h   kubernetes-ingress-controller        haproxytech/kubernetes-ingress:1.7.0   app.kubernetes.io/instance=kubernetes-ingress,app.kubernetes.io/name=kubernetes-ingress                   app.kubernetes.io/instance=kubernetes-ingress,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=kubernetes-ingress,app.kubernetes.io/version=1.7.0,helm.sh/chart=kubernetes-ingress-1.17.4
deployment.apps/kubernetes-ingress-default-backend   2/2     2            2           2d1h   kubernetes-ingress-default-backend   k8s.gcr.io/defaultbackend-amd64:1.5    app.kubernetes.io/instance=kubernetes-ingress,app.kubernetes.io/name=kubernetes-ingress-default-backend   app.kubernetes.io/instance=kubernetes-ingress,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=kubernetes-ingress,app.kubernetes.io/version=1.7.0,helm.sh/chart=kubernetes-ingress-1.17.4

 

Replicaset

NAME                                            DESIRED   CURRENT   READY   AGE    CONTAINERS                      IMAGES                                 SELECTOR                                                                                                               LABELS
replicaset.apps/kubernetes-ingress-64b5698b4b   2         2         2       2d1h   kubernetes-ingress-controller   haproxytech/kubernetes-ingress:1.7.0   app.kubernetes.io/instance=kubernetes-ingress,app.kubernetes.io/name=kubernetes-ingress,pod-template-hash=64b5698b4b   app.kubernetes.io/instance=kubernetes-ingress,app.kubernetes.io/name=kubernetes-ingress,pod-template-hash=64b5698b4b

 

Deploying Ingress Controller

Create a file: ing.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    haproxy.org/ingress.class: haproxy
  name: demo-restmon-ha-ingress
spec:
  rules:
  - host: "*.demo"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dx-restmon-restmon-demo
            port:
              number: 8080

Deploy using the following command: kubectl create -f ing.yaml
Create using the following command: kubectl create -f ing.yaml

Use the kubectl create command to create the ingress from the “ing.yaml” file.

Validate the Ingress Deployed

kubectl get ing -owide

NAME                      CLASS    HOSTS              ADDRESS          PORTS   AGE
demo-restmon-ha-ingress   <none>   *.demo             10.101.156.196   80      2d1h

In our machine, I have the ingress service deployed as below.

kubernetes-ingress                   NodePort       10.101.156.196   <none>        80:30125/TCP,443:32388/TCP,1024:30313/TCP   

 

So, ingress will allow information access via port 30125.

curl -X GET "http://localhost:30125/restmon/api/v1/version" -H "accept: */*" -H "Host: restmon.demo" -u admin -v
Enter host password for user 'admin':

* About to connect() to localhost port 30125 (#0)
*   Trying ::1...
* Connection refused
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 30125 (#0)
* Server auth using Basic with user 'admin'
> GET /restmon/api/v1/version HTTP/1.1
> Authorization: Basic YWRtaW46cGFzc3dvcmQ=
> User-Agent: curl/7.29.0
> accept: */*
> Host: restmon.demo
>
< HTTP/1.1 200
< set-cookie: JSESSIONID=8F6CF3D2EBFD379D516F675A31D19516; Path=/restmon/api; HttpOnly
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: 0
< x-frame-options: DENY
< content-type: text/plain;charset=UTF-8
< content-length: 22
< date: Sat, 16 Oct 2021 08:22:03 GMT
<
* Connection #0 to host localhost left intact
2.1.3 2021-09-08 10:0

Validate Version Deployed (via Browser)

Go to admin CRUD operations and navigate to the “version” API.

Broadcom Enterprise Software Academy - Employing RESTmon to Ingest Zabbix Data: Part 1

Once you execute, you should see a response like that below.

Broadcom Enterprise Software Academy - Employing RESTmon to Ingest Zabbix Data: Part 1

Now that RESTmon is up and running, you can begin to use it to feed information into DX Operational Intelligence. See the second part of this skill builder to learn how to use RESTmon to import data from Zabbix.

Broadcom Enterprise Software Academy - Employing RESTmon to Ingest Zabbix Data: Part 1