Automic Automation Cloud Integrations: AWS Glue Automation Agent

Instantly inherit the advanced capabilities of your enterprise solution, enabling you to deliver your digital transformation more quickly and successfully. This video explains the Automic Automation AWS Glue agent integration and its benefits. It presents its components and demonstrates how to install, configure, and use it.

Video Transcript

Overview of the Solution

Welcome to this video on the Automic Automation AWS Glue integration solution. In this video, we will explain the AWS Glue integration and what it brings to the Automic Automation user community.

AWS Glue is a serverless data integration platform that allows you to extract and transform data which is then loaded back to the data warehouse. This set of actions is also known as an ETL workflow:

• Extract
• Transform
• Load

That data is then available for analytics, machine learning, or application development purposes. Integrating Automic Automation with AWS Glue allows you to run AWS Glue jobs in your workspace from Automic Automation.

We will provide some technical insights so that you can clearly identify the integration components and understand the deployment sequence. We will focus on:

• The configuration of the agent.
• The design of the two core object templates: connections and jobs.
• A walkthrough demo.

The Role of Orchestration

Automic Automation plays a central role in orchestrating operations across multiple environments, including the cloud. Automic Automation synchronizes these processes with other non-cloud operations. By integrating AWS Glue, we can:

• Configure process automation centrally in Automic Automation.
• Trigger, monitor, and supervise everything in one place.
• Synchronize AWS Glue processes with all other environments routinely supported by Automic Automation.

In this setup, AWS Glue's role is reduced to executing the jobs. All other functions, especially those about automation, are delegated to Automic Automation. This means that we do not have to log in to the AWS Glue environment and manually refresh it; Automic Automation manages all the execution and monitoring aspects.

The Automic Automation integration provides a simplified view to run jobs in AWS Glue. Automic Automation lets us build configurations with intuitive interfaces like drag-and-drop workflows and supervised processes in simple dashboard tools designed natively for operations. Statuses are color-coded, and retrieving logs is done with a basic right-click.

From an operations perspective, Automic Automation highly simplifies the configuration and orchestration of AWS Glue jobs. Externalizing operations to a tool with a high degree of third-party integration means we can synchronize all cloud and non-cloud workloads using various agents and job object types. We can build sophisticated configurations involving:

• Multiple applications.
• Database packages.
• System processes like backups and data consolidation.
• File transfers.
• Web services.
• Other on-premise workloads.

Architecture and Deployment

A conventional architecture involves two systems: the Automic Automation host and a dedicated system for the agent.

The agent is configured with a simple INI file containing standard values: System, Agent Name, Connection, and TLS. When we start the agent, it connects to the engine and adds two new objects to the repository:

1. A connection object to store the AWS Glue endpoint and login data.
2. A job template designed to trigger AWS Glue jobs.

Let's assume we're automating for four instances of AWS Glue. We create a connection object in Automic Automation for each instance by duplicating the connection template for each of these instances. Lastly, we create an AWS Glue job in Automic Automation for each corresponding process in AWS Glue. The Automic Automation jobs include the connection object based on the target system.

When we execute the jobs in Automic Automation, it:

• Triggers the corresponding process in AWS Glue.
• Retrieves the successive statuses.
• Supervises the child processes in the cloud.
• Generates a job report.

In Automic Automation, these jobs can be incorporated into workflows and integrated with other non-cloud processes.

Step-by-Step Deployment Procedure

The procedure to deploy the AWS Glue integration is as follows:

1. Download the integration package from the marketplace. This package contains all the necessary elements.
2. Unzip this package, which produces a directory containing the agent, the INI configuration files, and several other items like the start command.
3. Update the INI file. AWS Glue is a standard Automic agent and requires at least four values to be updated:
   • The Agent Name.
   • The Automic System.
   • The JCP connection and TLS port.
   • The TLS certificate.
4. Start the agent. Once the agent is configured and started, new object templates are deployed.
5. Create connections. Use the template connection object and duplicate it for every AWS Glue instance you need to support. The connection object references the AWS Glue endpoint.
6. Create jobs. Use the AWS Glue template job to create the jobs you need. Match these Automic Automation jobs to the AWS Glue jobs, reference the connection object, and run them.

Once running, you are able to supervise the jobs and their children, generate logs, and retrieve statuses. The jobs can then be incorporated into non-cloud workflows.

Technical File Details

We unzip the package which creates a file system: agent/AWS glue/bin that contains the agent files. Based on the platform, we rename the agent configuration file UCXJCITX and set the minimum four values mentioned previously. Finally, we start the agent by invoking the JAR file via the Java command.

Demonstration: Configuration and Execution

In this demo, we will first create a connection object to your AWS Glue environment. Then we will focus on demonstrating how to create, execute, and supervise jobs.

Supported Job Types

There are various job types that the AWS Glue integration with Automic Automation supports:

• Run jobs: For defining jobs to get source data, process it, and write it to a target.
• Crawler jobs: For defining crawlers to search for and extract information from data sources to populate the AWS Glue data catalog.
• Trigger jobs: For defining triggers to start specific jobs and crawlers.
• Blueprint jobs: For defining blueprints to generate AWS Glue workflows.
• Workflow jobs: For defining workflows to run.

In this demo, we will show the start job run type because the functionality for other job types is similarly straightforward.

AWS Console Setup

1. Log in to the AWS Management Console and open the AWS Glue service.
2. From the navigation pane, go to ETL jobs. This section lets you create and run ETL workloads.
3. Under ETL jobs, you will typically see:
   • Visual ETL: Build ETL jobs using a visual drag-and-drop interface.
   • Notebooks: Develop Spark-based ETL using interactive notebooks.
   • Job run monitoring: Track job runs, execution status, logs, and run history.
4. For this demo, select visual ETL to design and manage glue ETL jobs. This page shows the existing glue jobs that are already configured in your AWS account.

Automic System Configuration

In the Automic system, we create connection and job objects.

• Endpoint: This is the most important field—the URL of the AWS Glue environment.
• Region: Specify the region in the AWS account where Glue resides.

Credential Methods

You can select from four types of credential methods:

1. AWS credentials file path: Define the credential profile name and the file location of the AWS credential file on the agent machine.
2. EC2 profile instance type: Allows you to connect to an EC2 VM within an AWS cloud application by defining the profile instance name.
3. Secret access key: Define an access key and the encrypted secret access key value.
4. External provider: Allows you to set up single sign-on (SSO) with SAML using either a service or an identity provider (such as Azure). You define the Tenant ID, Authentication URL, SAML username, SAML password, Principal ARN, Role ARN, and the Identity Provider.

In this demo, we use the secret access key credential method. If you are using a proxy, you can specify the proxy host name, port, username, and password in the proxy section.

Executing the Job

Once the connection object is saved, we create a run job.

1. On the attributes page, select the agent that corresponds to the name configured in the INI file.
2. On the start job run page, select the connection object we created.
3. Define the job name (type it in or use the picker field).
4. Specify parameters: Define payload parameters in JSON format. You can select between none, JSON, and JSON file path.

Once all parameters are set, save and execute the job.

Monitoring and Reporting

Switch to the executions view. The details pane shows the remote status field, which tracks the job status in the target environment. In our case, the execution was successful. The details panel also shows an object variable called job run ID, which helps identify the specific job execution for further analysis.

Reports

• Report 1: Contains all information the target system sends to Automic Automation. This is presented in a structured JSON output including execution details and results.
• Verification: Make a note of the run ID (e.g., ending with EF4C36).
• AWS Verification: Switch back to the AWS Glue console, search for the Glue job name, and double-click it. Open the runs tab to view the history. You will see that the job run ID returned by AWS Glue is captured in the Automic Automation output.
• Agent Log (P log): Shows all the agent's actions step-by-step, listing all parameters used and the response received from the target system.

This concludes the demo on how Automic Automation can integrate with AWS Glue to run, execute, and monitor jobs. Thank you for watching this video.

Note: This transcript was generated with the assistance of an artificial intelligence language model. While we strive for accuracy and quality, please note that the transcription may not be entirely error-free. We recommend independently verifying the content and consulting with a product expert for specific advice or information. We do not assume any responsibility or liability for the use or interpretation of this content.