Automic Automation Cloud Integration: DBT

Broadcom's DBT Automation Agent lets you easily execute DBT jobs. You can also monitor and manage jobs with your existing enterprise workload automation and perform other cloud-native activities. You instantly inherit the advanced capabilities of your enterprise solution, enabling you to deliver your digital transformation more quickly and successfully.

Video transcript

1. Overview of the integration

Data Build Tool (DBT) is an open-source command-line tool used by analysts and engineers to transform data within warehouses. Integrating it with Atomic Automation allows users to trigger, monitor, and synchronize DBT transformations from a centralized orchestration platform.

Key value propositions

• Centralized orchestration: Manage cloud-based DBT processes alongside traditional, non-cloud operations.

• Delegated automation: DBT handles execution, while Atomic Automation manages scheduling, monitoring, and error handling.

• Simplified operations: Users can avoid manual logins to the DBT environment, using Atomic’s intuitive interface instead.

• Visual management: Benefit from drag-and-drop workflows, color-coded statuses, and right-click access to logs.

2. Technical architecture

The integration relies on a standard agent-based architecture to facilitate communication between the Atomic Automation engine and the DBT system.

Core components

• Atomic Automation host: The central engine managing the workflows.

• Dedicated agent system: A system running the DBT agent, configured via an INI file.

• Connection objects: Store the DBT endpoint and authentication credentials.

• Job templates: Pre-configured objects used to trigger specific DBT processes.

3. Deployment sequence

To deploy the integration, follow these structured steps:

• Download and extract: Acquire the integration package from the Marketplace and unzip it to create the directory structure (e.g., agent/dbt/bin).

• Configuration: Rename and edit the agent configuration file (UCXJCX). You must update at least four values:

1. 1. Agent name: The unique identifier for the agent.

   2. System name: The name of the Atomic Automation (AE) system.

   3. Connection details: The JCP host and TLS port.

   4. Security: Path to the TLS certificate.

• Start the agent: Invoke the .jar file using a Java command. Upon connection, the agent automatically deploys the necessary Connection and Job templates to the repository.

• Create objects: Duplicate the connection templates for each DBT instance and create job objects for every corresponding DBT process.

4. Configuration details

Once the agent is active, you must configure the specific objects required to run jobs.

The connection object

The Connection object acts as the bridge to your DBT environment.

Key fields include:

• Endpoint: The URL of the DBT environment.

• Token: The authentication token for API access.

• Proxy settings: (Optional) Includes Proxy Host, Port, Username, and Password, along with a "No Proxy Hosts" field for bypass rules.

The trigger job object

The Job object defines how a transformation is executed. You must first assign the correct Agent and Connection object in the attributes. Then, define the Execution Type:

Parameters mode:

• Trigger mode: Choose to trigger by Job ID or Job Name.

• Identifiers: Account ID, Job Name, or Job ID.

• Context: The "Cause" field explains why the job was triggered.

• Git integration: (Optional) Define the Git SHA (commit hash) or Git Branch.

JSON payload mode:

• Input the parameters directly as a JSON string or provide a File Path to a JSON file.

5. Execution and monitoring

After saving and executing a job, Atomic Automation provides several tools for supervision:

Monitoring features

• Remote status: Located in the Details pane, this tracks the real-time status of the job in the DBT cloud.

• Object variables: The system captures variables like run_id and run_f to identify specific executions.

• Automated reporting:

  • Structured report: Displays the JSON response from DBT, including execution details.

  • Agent log (P log): Provides a step-by-step audit trail of agent actions and the raw response from the target system.

Note: The run_id captured in Atomic Automation will match the Run History ID in the DBT Cloud Console, ensuring full traceability across systems.

Note: This transcript was generated with the assistance of an artificial intelligence language model. While we strive for accuracy and quality, please note that the transcription may not be entirely error-free. We recommend independently verifying the content and consulting with a product expert for specific advice or information. We do not assume any responsibility or liability for the use or interpretation of this content.