Automic Automation Cloud Integrations: Databricks Agent Integration

Video Transcript

Welcome to this video on the Automic Automation Databricks integration solution. In this video, we will explain the Databricks Cloud integration and what it brings to the Automic Automation user community.

Databricks is a popular data platform used for ETL processing, analytics, and machine learning. Automating execution from within Databricks creates an additional island of automation for IT operations to manage. The Automic Databricks agent allows you to integrate Databricks jobs with your existing enterprise workload automation easily. You instantly inherit the advanced capabilities of your enterprise solution to deliver your digital transformation quicker and more successfully.

By extending your existing enterprise automation to Databricks, you maintain end-to-end visibility, regain centralized command and control, enable alerting, SLA management, reporting, and auditing. Integrating Automic Automation with Databricks allows you to run Databricks jobs in your workspace from Automic Automation.

We'll provide some technical insights so that the integration components are clearly identified and the deployment sequence is understood. We'll focus on the configuration of the agent and the design of the two core object templates: connections and jobs. Finally, we'll run through a demo.

Automic Automation plays a central role in orchestrating operations across multiple environments, including the cloud. Automic Automation synchronizes these processes with other non-cloud operations. By integrating Databricks, we can configure process automation centrally in Automic Automation and then trigger, monitor, and supervise everything in one place.

Databricks processes can then be synchronized with all other environments routinely supported by Automic Automation. Databricks' role is reduced to executing jobs, while all other functions, especially those pertaining to automation, are delegated to Automic Automation. This means that we don't have to log into the Databricks environment and manually refresh it. Automic Automation manages all execution and monitoring aspects.

The Automic Automation integration provides a simplified view of Databricks jobs. Automic Automation lets us build configurations with intuitive interfaces like drag-and-drop workflows and supervised processes in simple dashboard tools designed natively for operations. Statuses are color-coded, and retrieving logs is done with a basic right-click.

From an operations perspective, Automic Automation highly simplifies the configuration and orchestration of Databricks jobs. Externalizing operations to a tool with a high degree of third-party integration means we can synchronize all cloud with non-cloud workloads using various agents and job object types. We can build sophisticated configurations involving multiple applications, database packages, system processes like backups and data consolidation, file transfers, web services, and other on-premise workloads.

A conventional architecture involves two systems: the Automic Automation host and a dedicated system for the agent. The agent is configured with a simple INI file containing standard values: system, agent name, connection, and TLS. When we start the agent, it connects to the engine and adds two new objects to the repository: a connection object to store the Databricks endpoint and login data, and a job template designed to trigger Databricks jobs.

Let's assume we're automating for four instances of Databricks. We create a connection object in Automic Automation for each instance by duplicating the connection template for each of these instances. Lastly, we create the Databricks jobs in Automic Automation for each corresponding process in Databricks. The Automic Automation jobs include the connection object based on the target system. When we execute the jobs in Automic Automation, it triggers the corresponding process in Databricks. We're able to retrieve the successive statuses and finally generate a job report in Automic Automation. This job can be incorporated into workflows and integrated with other non-cloud processes.

The procedure to deploy the Databricks integration is as follows:

First, we download the integration package from Marketplace. This package contains all the necessary elements. We unzip this package, which produces a directory containing the agent, the INI configuration files, and several other items like the start command. We use the appropriate INI file for our specific platform. Databricks is a standard Automic agent. It requires at least four values to be updated: agent name, Automic system, JCP connection, and TLS port, and finally, TLS certificate. When the agent is configured, we start it. New object templates are deployed.

We create a connection to every Databricks instance we need to support. For this, we use the template connection object, which we duplicate as many times as needed. The connection object references the Databricks endpoint. Finally, we use the Databricks template jobs to create the jobs we need. We match these Automic Automation jobs to the Databricks jobs, reference the connection object, and run them. We're able to supervise the jobs, generate logs, and retrieve the statuses. The jobs can then be incorporated into non-cloud workflows.

We install, configure, and start an agent to deploy the Databricks integration. The agent is included in the Databricks package, which we download from Marketplace. We unzip the package, which creates a file system agent: SL-Databricks/bin that contains the agent files.

Based on the platform, we rename the agent configuration file to ucx.jci.idx and set a minimum of four values: the agent name, the AE system name, the host name and port connection to the automation engine’s JCP, and finally, the directory containing the TLS certificate. Finally, we start the agent by invoking the JAR file via the Java command. The agent connects to the AE and deploys the object templates needed to support the integration: the connection object and the Databricks job templates.

Automic can connect to Azure Databricks and AWS Databricks cloud types. In our demo, we will create a connection object for Azure Databricks. Once we have established the connection to the Azure Databricks environment, we'll create a Databricks job. Finally, we'll execute and supervise this job.

The Azure Databricks console offers a variety of features accessible through the left navigation pane. From the navigation pane, select Job Runs to view multiple jobs that have already been created. When you open one, you can view the details of that specific job.

Moving to the Automic system, we create a connection object with specific inputs to connect to Databricks. We must specify the cloud type—either AWS or Azure. Regardless of which cloud type we select, we must also specify the workspace URL, indicating where the Databricks service resides.

For Azure Cloud, we must select an authentication type, choosing from:

• Service Principal Type
• Token from File Type
• Personal Access Token Type

For AWS Cloud, we must specify the workspace URL and select the appropriate authentication type.

Once the connection object is defined, we create Databricks jobs, including Run Jobs and Start/Stop Cluster Jobs.

Finally, after executing a Run Job, we navigate to the Executions View in Automic Automation to verify job success. The Details Pane displays execution logs, object variables, and the Databricks run ID.

That wraps up the demo on how Automic Automation integrates with Databricks to run, execute, and monitor Databricks jobs. Thank you for watching!

Note: This transcript was generated with the assistance of an artificial intelligence language model. While we strive for accuracy and quality, please note that the transcription may not be entirely error-free. We recommend independently verifying the content and consulting with a product expert for specific advice or information. We do not assume any responsibility or liability for the use or interpretation of this content.