Automic Automation: Delivering Data Pipelines to Production Using Multi-Cloud Orchestration

Read this white paper for an overview of how Automic Automation’s capabilities provide the tools you need to orchestrate processes in multiple platforms.

A Workflow orchestrates the following use case:

• A job generates data from a local application (for example, Oracle or SAP).

• This data should  undergo certain transformation processes. To make the local data available to the cloud solutions that will transform it, it is first uploaded to cloud-based store tools, Google Cloud Storage and Azure Blob.

• The data is transformed through Google BigQuery and Azure DataBricks.

• The data is sent to a third-party application in a Kubernetes cluster for inventory calculations.

All Automic Automation Agent Integrations operate the same way. Learn how to download, install, configure and start the Automic Cloud Integration Agents.

Azure Blob Storage is Microsoft Azure’s Cloud storage solution and the standard storage cloud solution for many Automic Automation users. Customers of all sizes and industries use Azure Blob Storage to store (upload and download) unstructured data such as text and binary data, to move and monitor it, to delete it and so forth.

In our use case, the data generated by a local application needs to be uploaded to the Cloud for Azure Databricks. Before this can happen, the data must be made available in the cloud. The Workflow contains a Blob Upload Job that does this.

Azure DataBricks is Microsoft Azure’s Cloud data management solution. It brings data lakes (large data volumes) to lakehouses (systems built to exploit these volumes). Databricks is popular in certain circles: Data science, data engineering and Machine Learning.  

Databricks has a proprietary job processing tool. Automic triggers these jobs, or can submit them independently via JSON, bypassing Databricks.

The data needs to undergo transformation before being transferred for inventory analysis.  

One of the major challenges that Automic Automation teams encounter when designing cloud automation processes is how to authenticate to the target cloud systems This video explains the various methods applicable to each integration, giving special attention to the Azure Service Principal, which applies to all Azure integrations.

Google Cloud Storage and AWS S3 are industry-leading, object storage services that are the standard for many Automic Automation users. Customers of all sizes and industries use them to store and protect any amount of data for a range of use cases, such as big data analytics, backup and restore, archive, enterprise applications, and so on.

In our use case, the data needs to be uploaded to the Cloud for Airflow and BigQuery. 

Airflow is an open-source, Apache Cloud-based workflow automation solution, which was incorporated into the Google Cloud Platform. Both execute Python-based jobs made of sequential tasks. For Automic Automation, this means triggering DAG jobs, capturing the status of the individual tasks and of the overarching DAG Job, and reporting.

In our use case, the data has been uploaded and an auxiliary process is now triggered in the Google Cloud Composer. Airflow and GCC use the same Automic Agent.

Google BigQuery is a data warehousing service for business intelligence, particularly suitable for AI. BigQuery is a major player in Big Data analytics. It includes native job capabilities: Loading, exporting, querying or copying data. Jobs are coded with a mixture of languages, like python and SQL. They are then listed as individual objects in the console, as Transfer Configs.

In our use case, the data needs to undergo a process of transformation before being transferred for inventory analysis.

One of the major challenges that Automic Automation teams encounter when designing cloud automation processes is how to authenticate to the target cloud systems. This video explains the various methods applicable to each integration, giving special attention to the  Service Account Key, which is central to authenticating with Google Cloud applications. 

One of the major challenges that Automic Automation teams encounter when designing cloud automation processes is how to authenticate to the target cloud systems. This video explains the various methods applicable to AWS S3.

Kubernetes is an open-source container system for scaling software applications. It assembles virtual or bare metal systems into a cluster which can run containerized workloads. Individual software components (processes, databases, web servers...) are isolated in individual virtual pods, which can be up or downscaled to optimize capacity management.

In our use case, this Job builds a gateway into the Kubernetes environment so as to trigger processes in the clusters.

All the pieces are ready now, so we can execute the Workflow.

In our use case, we execute the Workflow and explain its progress. We look at and explain the reports and Agent logs.