FAQs on the Configuration of Kubernetes and Automic Automation When Moving to AAKE

Question

Answer

Preparing for the AAKE deployment

How do I know how many replicas I need for my AAKE deployment?

The same sizing guidelines apply for AAKE and on-prem, and the number of processes required for daily work should be configured via values.yaml before deployment:

spec:

awiReplicas: 2
cpReplicas: 0
jcpRestReplicas: 1
jcpWsReplicas: 2
jwpReplicas: 2
wpReplicas: 4

Sizing guidelines:

https://docs.automic.com/documentation/webhelp/english/AA/21.0/DOCU/21.0/Automic%20Automation%20Guides/Content/Installation_Common/install_AWA_ARA_sys_requirements_sizing.htm   

How do I request and limit resources for each AAKE pod/replica within my Kubernetes cluster?

You can define that in the values.yaml file.

This is also described in the Sizing guidelines:

https://docs.automic.com/documentation/webhelp/english/AA/21.0/DOCU/21.0/Automic%20Automation%20Guides/Content/Installation_Common/install_AWA_ARA_sys_requirements_sizing.htm

Do I need the Automic Automation Helm plugin?

Yes, the plugin requires Helm version 3 and is used to monitor the progress of the deployment or upgrade. It is also mandatory when upgrading to a newer AAKE version, since it pauses the installation until the DB has been backed up.

Since not all commands will work from a Windows CLI, you should use the plugin with a Linux CLI.

https://docs.automic.com/documentation/webhelp/english/AA/21.0/DOCU/21.0/Automic%20Automation%20Guides/Content/Installation_Containers/containers_HelmPlugin.htm 

How do I access the Automic server from outside the Kubernetes cluster?

AWI, JCP WS and JCP REST require HTTP(S) endpoints that are exposed through an Ingress (fulfilled by an Ingress Controller/HTTP(S) Load Balancer).

https://docs.automic.com/documentation/webhelp/english/AA/21.0/DOCU/21.0/Automic%20Automation%20Guides/Content/Installation_Containers/containers_JCP_REST_Using_Ingress.htm

CPs need to be exposed via a TCP Load Balancer/TCP Proxy.

https://docs.automic.com/documentation/webhelp/english/AA/21.0/DOCU/21.0/Automic%20Automation%20Guides/Content/Installation_Containers/containers_Reaching_CP_Endpoint.htm 

How do I configure the Ingress? 

The AAKE install operator can automatically configure ingresses for NGINX. Make sure enable: true under the ingress: section in values.yaml.
Ingress configuration is different across cloud providers, although some allow the use of NGINX as well as cloud provider specific ingress service.

https://docs.automic.com/documentation/webhelp/english/AA/21.0/DOCU/21.0/Automic%20Automation%20Guides/Content/Installation_Containers/containers_JCP_REST_Using_Ingress.htm 

Do I need certificates for AAKE?

Yes, the TLS agents will perform the TLS handshake with the HTTPS Load Balancer before connecting to the JCPs inside the cluster. A private key and certificate need to be configured at the Ingress level, but no additional configuration is required for the JCP, as is the case for on-prem installations.

https://docs.automic.com/documentation/webhelp/english/AA/21.0/DOCU/21.0/Automic%20Automation%20Guides/Content/Installation_Containers/containers_InstallingPreparing.htm#link5 

How can I configure my own Automic system name?

Persistent configuration (like system name for server and AWI) should be set via environment variables in values.yaml:

environment:

  AUTOMIC_GLOBAL_SYSTEM: AUTOMIC

  AUTOMIC_SYSTEM_NAME: AUTOMIC

How do I enable SAML for AAKE?

Persistent configuration (like enabling SSO) should be set via environment variables in values.yaml:

environment:

  AUTOMIC_SSO_SAML_ENABLED: true

Can I use only one tablespace for my (managed) DB?

For managed cloud DB services you may not have the ability to use different tablespace names, which is why Automic Automation V21 supports configuring both names to the same if needed.
For AAKE, the tablespace name can be configured in the DB secret before deployment.

How do I create a DB secret?

kubectl create secret generic ae-db \

--from-literal=host=aut-db.eu-central-1.com \

--from-literal=vendor=postgres \

--from-literal=port='5432' \

--from-literal=user=oab \ 

--from-literal=db=ae \

--from-literal=password=automic \

--from-literal=data-tablespace-name=pg_default \

--from-literal=index-tablespace-name=pg_default \

--from-literal=additional-parameters="connect_timeout=10 client_encoding=LATIN9"

How do I create a secret for the client 0 user?

kubectl create secret generic client0-user \

  --from-literal=client='0' \

  --from-literal=user='ADMIN' \

  --from-literal=department='ADMIN' \

  --from-literal=password='admin'

How do I create a TLS secret?

If you want to use the automatically generated Ingresses that are configured for an NGINX Controller, a TLS secret containing the private key and certificate is required.

kubectl create secret tls certificate-tls-secret --key private_key.pem --cert certificate.pem

Can I use ZDU to upgrade from v12.3 to AAKE v21?

No, you must not use ZDU in AAKE-context. There will be a downtime when switching from an on-prem installation to the AAKE deployment. 

NB: Since ZDU isn’t technically prevented by AAKE V21, the admin has to be aware of this fact

Will the Automic Proxy work with AAKE?

Yes, with v21, the Proxy Client connects to the JCP via TLS. The TLS agents can then connect to the Proxy Server. The communication between the 2 Proxy components has not changed.

https://docs.automic.com/documentation/webhelp/english/AA/21.0/DOCU/21.0/Automic%20Automation%20Guides/Content/Proxy/about_proxy.htm 

Can I have an AAKE deployment without CPs?

By default, the CP replicas are set to 0 in the values.yaml file. 

CPs are only required if at least one of the two cases is true:

• non-TLS agents can’t be connected to the JCP via the TLS Gateway in CP mode
• OS CallAPIs have to be used (Unix, Windows, z/OS)

How can I connect my old/non-TLS agents (<V21.0) to AAKE 21.0?

This can be done via the following two ways:

• Via a TLS Gateway (with activated CP mode) - this is the recommended final way, no CP pods are needed anymore in this case, even you still use old/non-TLS agents
• Via old CPs with help of a TCP load balancer that has to be made available in your AAKE cluster. The load balancer acts as an externally accessible connection point for the agents and distributes requests to the CPs inside the AAKE cluster - this is not an recommended final way, but it is a good way to test and gain experiences with AAKE by reducing its  complexity by omitting the TLS Gateway in the first steps. Keep in mind that you will not be able to perform file transfers between TLS and non-TLS agents, this works via a TLS Gateway only. The needed TCP load balancer is not an Automic product or shipped along with it, it is part of your  Kubernetes environment.

https://docs.automic.com/documentation/webhelp/english/AA/21.0/DOCU/21.0/Automic%20Automation%20Guides/Content/Installation_Containers/containers_Reaching_CP_Endpoint.htm 

What are some best practices when migrating from on-prem to the container-based deployment?

1. Make sure the database is ready and can be accessed from the Kubernetes cluster
2. Check the ucsrv.ini and AWI config and properties files for settings that you will require (system name, SSO, CORS, ...)
3. Decide if an existing client 0 user should be used to customize the AAKE installation
4. If old agents still need to be able to connect to the cluster 
1. Connect to the CP -> adapt agent ini files to point to the TCP Load Balancer address
2. Using the v12.3 Automic Proxy -> adapt Proxy ini file to point to the TCP Load Balancer address
3. Using the TLS Gateway to connect to the JCP

What about Analytics when deploying AAKE?

The way how Analytics is handled depends on your installation scenario: 

• New installation - If you start with an initial installation on AAKE 21.0, Analytics can be installed optionally via a switch in the values.yaml file (analytics.enable: true | false). In this case an Analytics pod will be deployed. The connection between AE and Analytics is configured/set up automatically.
• Migration/Upgrade - If you migrate from AA 12.3 to AAKE 21.0, Analytics will stay outside the Kubernetes cluster as on-premise installation, and you have to upgrade it accordingly. In this case you have to configure the connection between Analytics (on-premise) and the AE (in Kubernetes cluster).

https://docs.automic.com/documentation/webhelp/english/AKE/21.0/DOCU/21.0/Automic%20Automation%20Guides/Content/Installation_Containers/containers_Installing.htm#link6

https://docs.automic.com/documentation/webhelp/english/AKE/21.0/DOCU/21.0/Automic%20Automation%20Guides/Content/Installation_Containers/containers_InstallingPreparing.htm#ConfiguringAnalyticsOnPrem_AAKE 

Are there any special things to consider when migrating from an on-prem to a managed DB?

In versions prior to v21, minimum 2 tablespaces (ae_data, ae_index) were required for an installation.

Managed databases usually do not allow users to create tablespaces, so the default DB instance tablespace has to be used. This is possible with v21, but you need to prepare for this when doing the DB migration.

Cloud providers might offer migration services and tools, for Google Cloud this guide describes how to migrate to a Cloud SQL PostgreSQL DB https://cloud.google.com/database-migration/docs/postgres/quickstart 

Are TLS secured database connections supported?

For standard AE with PostgreSQL yes, see https://docs.automic.com/documentation/webhelp/english/AA/21.0/DOCU/21.0/Automic%20Automation%20Guides/Content/Installation_Common/PreparationSteps/PrepareAEDB_PostgreSQL.htm#link9

For AAKE this is not (yet) supported

Are sticky sessions required to access AWI via Load Balancer?

Yes, this is included in the documentation and the configuration is Cloud provider specific.

https://docs.automic.com/documentation/webhelp/english/AA/21.0/DOCU/21.0/Automic%20Automation%20Guides/Content/Installation_Manual/AWI/ConfiguringAWI.htm?Highlight=sticky%20session#link20 

On a running AAKE system

How do I set logging / trace levels for AE?

Via the ae-properties configmap. Setting this will restart all the AE pods.

data:

  AUTOMIC_TRACE_TRC03: '1'

https://docs.automic.com/documentation/webhelp/english/AA/21.0/DOCU/21.0/Automic%20Automation%20Guides/Content/Installation_Containers/containers_ConfigureContainer.htm?Highlight=configmap

How do I change the default AWI configuration?

Via the awi-properties configmap. Settings that were previously in uc4config.xml, colors or configuration properties can be modified. The AWI pods will restart automatically. 

data:
  AUTOMIC_SESSION_COLORS: '#21DE4A,#DE6621,#133AD4' 

How do we get customer AAKE logs now that we no longer create standard log files?

Customer already knows how to get these, but normally log management will be in place, otherwise kubectl logs <pod name> can be used.

How to do a COLD START?

Via the ae-properties configmap, this will restart all the AE pods.

data:
  AUTOMIC_GLOBAL_STARTMODE: 'COLD'

How to do a full system restart?

A Google search for “How do I restart all pods” will provide multiple answers, including kubectl -n <namespace> rollout restart deploy

Cloud Provider/Platform specifics

Do we have any instructions / best practices on how to deploy AAKE to AWS?

Yes:

https://academy.broadcom.com/hubfs/ESD/ESD_Academy/ESD_FY21_Academy/ESD_FY21_Academy_Files/ESD_FY21_Academy_Files_AIOps/Installing%20Automic%20Automation%20Kubernetes%20Edition%20on%20GCP%20(v1.1).pdf?hsLang=en

Does AAKE work on AWS Fargate?

Yes, see above link.

Do we have any instructions / best practices on how to deploy AAKE to Azure?

Yes:

https://academy.broadcom.com/hubfs/ESD/ESD_Academy/ESD_FY21_Academy/ESD_FY21_Academy_Files/ESD_FY21_Academy_Files_AIOps/Installing%20Automic%20Automation%20Kubernetes%20Edition%20on%20Microsoft%20Azure%20(v1.1).pdf?hsLang=en

Do we have any instructions / best practices on how to deploy AAKE to GCP?

Yes:

https://academy.broadcom.com/hubfs/ESD/ESD_Academy/ESD_FY21_Academy/ESD_FY21_Academy_Files/ESD_FY21_Academy_Files_AIOps/Installing%20Automic%20Automation%20Kubernetes%20Edition%20on%20GCP%20(v1.1).pdf?hsLang=en

Does AAKE work with OpenShift?

Although OpenShift is based on Kubernetes, it provides an additional PaaS management layer that has additional requirements. In theory, AAKE should work, however we consider official support for OpenShift out of scope as we focus on core Kubernetes, either self managed or managed Kubernetes services by the leading cloud providers.

Does AAKE work with Docker Swarm (or Mesos + Marathon)?

Kubernetes is the leading container orchestration platform that sees widespread adoption. As such we standardize on Kubernetes and the Kubernetes ecosystem.

We currently have no plans to support any of the alternative container orchestration platforms.

Does AAKE work on a Rancher Kubernetes cluster?

Since the Rancher Kubernetes Engine (RKE) uses the basic Kubernetes distribution, there is no reason why it can’t work. But any additional services required for cluster operations are not in scope of AAKE support and are the responsibility of the customer.

Is Google’s Cloud SQL Auth Proxy supported with AAKE?

No, at the moment it is not possible to configure the Cloud SQL Proxy to run as a sidecar container with AAKE. In order to use AAKE in GKE with a Cloud DB, the database instance has to be configured to use a private IP address.

General questions

What databases does AAKE support?

PostgreSQL and Oracle. This includes managed DB offerings from the leading cloud providers. Refer to compatibility matrix for exact versions:

https://downloads.automic.com/tools/compatibility_matrix?compatibility-mode=component&offering_version_id=1601569640847&lifecycle_entity_id=1601569166697&component_id=1409731986667&version_id=1585553688489)

Why doesn’t AAKE support MS SQL?

In fact, when running the AE on Linux, this was never supported. Only when running AE on Windows, MS SQL is supported as database. We are monitoring demand for MS SQL in context of AAKE and “regular” AE deployments. If there’s enough demand this may change in a future version.

Do we support Istio as a service mesh?

The number of tools and services that can be used with Kubernetes is countless and we can’t possibly try them all out. This does not mean they won’t work, but it’s up to the customer to set them up for AAKE.

Can you recommend any tools for log management?

Cloud providers often have these in place, but so far we’ve heard  customers using tools like Kibana or Splunk.

What are some beginner friendly open-source tools to try out with AAKE while learning about Kubernetes? 

Microk8s https://microk8s.io/ 

Lens https://k8slens.dev/ 

K9s https://k9scli.io/ 

Can I use a local/private repository instead of GCR to deploy AAKE?

Yes, it is possible to download the images before the deployment using docker pull or tools like Skopeo. The values.yaml file needs to be adjusted to point to the new repository and a new pull secret with the credentials for this repo has to be created.

How can I use Kubernetes autoscaling (horizontal pod autoscaler) with AAKE?

This is a good place to start (generic):

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/

The managed K8s services from the leading cloud providers, may provide their own implementation: 

https://cloud.google.com/kubernetes-engine/docs/how-to/horizontal-pod-autoscaling

Use HPA in combination with Prometheus for scaling using custom metrics:

https://towardsdatascience.com/kubernetes-hpa-with-custom-metrics-from-prometheus-9ffc201991e

Automic provides custom metrics for use with Prometheus:

https://docs.automic.com/documentation/webhelp/english/all/components/DOCU/21.0/REST%20API/Automation.Engine/index.html?overrideUrls=../Automation.Engine/swagger.json,../Analytics/swagger.json,../Infrastructure.Manager/swagger.json#/system%20metrics/allPrometheusMetrics

Which cluster permissions are required when deploying AAKE?

1. The Custom Resource Definition (CRD) for AutomicAutomation is created via Helm and requires cluster-scope write access https://helm.sh/docs/topics/rbac/
2. The AutomicAutomation Custom Resource (CR) is Namespaced, not Cluster-scoped and is used to store information relevant to the deployment (version, status, ...)
3. A Service Account (automic-operator-sa) is required for the Install Operator (namespace scope)
4. There are 2 Roles and Role Bindings (namespace scope) required for the installation, one for the operator and one for the CR:
1. aa-operator-role:
   Resources secrets, configmaps, pods

            Verbs get, list, watch, create, update, patch, delete

            Api Groups ''

            
            Resources deployments, jobs, services, ingresses

            Verbs get, list, watch, create, update, patch, delete

            Api Groups apps, extensions, batch, networking.k8s.io, ''

1. aa-operator-cr-role:

                       Resources automic-automations

                       Verbs get, list, watch, patch

                       Api Groups broadcom.com

Troubleshooting

Why can’t DBLoad connect to the DB during AAKE deployment?

1. The address/host where the DB can be reached has one of these formats, depending if it is running within the Kubernetes cluster or not and needs to be configured in the same db secret that is configured in values.yaml

• DB Server in a Kubernetes cluster:

      host: <db-server-service-name>.<namespace>.svc.cluster.local

• DB Server external:

      host: <db-server-domain-or-ip-address>

https://docs.automic.com/documentation/webhelp/english/AA/21.0/DOCU/21.0/Automic%20Automation%20Guides/Content/Installation_Containers/containers_InstallingPreparing.htm#link8 

2. The DB port is configured in the db secret and is not blocked by a Firewall for example.

3. There is a DNS service running in the cluster (kube dns).

My NGINX ingress controller has following error: “ingress does not contain a valid IngressClass", what is causing this?

On Aug 24, NGINX v1.0.0 was released, see release notes

https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.0.0

This release includes a breaking change, where an IngressClass must be specified on your ingress definitions.

Add following to the annotations: section of your ingress yaml file(s)

kubernetes.io/ingress.class: "nginx"

Why do I get this error when trying to create the Automic image pull secret?
“error: exactly one NAME is required, got 5”

If you create the secret using Windows PowerShell, the double quotes in the command have to be replaced by single quotes.

For example:

PS C:\Users\ob685245> kubectl create secret docker-registry test-automic-image-pull-secret --docker-server=gcr.io --docker-username=_json_key --docker-password='$(cat ./automic-image-pull-secret.json)' --docker-email=broadcom-com@esd-automic-saas.iam.gserviceaccount.com