Streamlining Airflow Workflows with Enterprise Automation

What was released?

Airflow integrations are now available for AutoSys, Automic Automation, and dSeries. Apache Airflow is an open-source workflow management platform used to develop, schedule, and monitor batch-oriented processing. Airflow is also available through a managed offering from the major cloud vendors. Automating execution from within Airflow creates an additional island of automation for IT Operations to manage.

Automation by Broadcom allows you to easily integrate directed acyclic graphs (DAGs) - a collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies - with your existing enterprise workload automation solution. Not only does Automation by Broadcom integrate with the open source Apache Airflow, we also integrate with the managed offerings from cloud vendors including Google Cloud Composer, Amazon Managed Workflows for Apache Airflow (MWAA), and Azure Data Factory Managed Airflow.

What do the integrations do?

They easily integrate Airflow DAGs with your existing Broadcom enterprise workload automation. You instantly inherit the advanced capabilities of your enterprise solution to deliver your digital transformation quicker and more successfully.

By extending your enterprise automation to Airflow you maintain end-to-end visibility, regaining centralized command and control, alerting, SLA management, reporting, and auditing.

With the Airflow integration, you can:

• Schedule execution of Airflow DAGs
• Monitor to completion of Airflow DAGs
• Retrieve Airflow DAG logs for problem diagnosis
• Restart Airflow DAG processing
• Alert on the failure of Airflow DAG processing
• Create dependencies between Airflow DAG executions
• Incorporate Airflow DAG processing in existing enterprise automation workflows including dependency enforcement

What are the scheduling limitations of Airflow?

While it may be easy for developers to get started with Airflow, many challenges can arise once applications are handed over to production. This is particularly true for teams in large enterprises.

When running applications in production environments, teams need to ensure adherence to standards and policies and support ongoing audits. Airflow presents many limitations that make these day-to-day operations difficult.

Limited scheduling options

Airflow’s scheduling capabilities are too limited to address the business needs of most enterprises. To schedule DAGs in Airflow, teams are limited to date/time criteria. Most business applications require much more scheduling sophistication.

Isolated, complex operation

With Airflow, there is no visibility into corporate-level calendars. That means something that should be simple, such as preventing workloads from running on holidays, is not very simple. This type of logic is managed at the individual DAG level. Any time calendaring changes need to be made, teams would need to make those changes individually for each DAG that was affected.

Even simple changes to a DAG schedule can be overly complicated, potentially requiring either the renaming of the DAG or editing data in the database. Further, within most organizations, multiple development groups will employ Airflow. Each deployment will have its own user interface. Consequently, teams will need to use a distinct UI for each Airflow deployment, and for any other tools being employed.

Limited reporting and visibility

In Airflow, reporting and auditing are virtually non-existent. Any limited analytics that are captured and displayed are at the individual DAG level. Additionally, you can’t view a failed workflow and quickly determine what the business impact is for any downstream work.

Limited capabilities and controls

When using Airflow, you can’t easily have dependencies between DAGs. Effectively, all dependencies need to be defined within a single DAG. With Airflow, you can’t easily kill a job, implement a one-time override, define priorities, or take advantage of features like virtual resources.

What are the benefits of integrating Airflow with Broadcom’s Automation Solutions?

Quite simply, Airflow lacks many of the standard, out-of-the-box capabilities Broadcom users have relied upon and grown accustomed to. Despite these limitations, development teams may still opt to use Airflow, and, indeed, various groups are already deeply invested in Airflow. That’s why the integration capabilities of Automation by Broadcom are so valuable.

There are many benefits to extending enterprise automation to cloud processing. With Automation by Broadcom, you can effectively establish a “manager of managers,” employing a unified tool to manage several disparate cloud scheduling and workflow orchestration solutions and get through end-to-end business process visibility.

Through these integrations, organizations get the best of both worlds. Development teams can use Airflow if they choose while operations teams get the power of enterprise-grade schedulers.

Where can I learn more about Broadcom’s cloud integrations for workload automation?

Broadcom’s Automation Marketplace makes it easy to browse and search for cloud integrations for Automation by Broadcom as they become available. There you can find information on all the Airflow integrations from Broadcom including the open-source Apache Airflow, Google Cloud Composer, Amazon MWAA, and Azure Data Factory Managed Airflow.

Visit Automation Marketplace for details on integration features, links to extended TechDocs, and the option to download the software.