<img height="1" width="1" style="display:none;" alt="" src="https://px.ads.linkedin.com/collect/?pid=1110556&amp;fmt=gif">
Skip to content
    September 18, 2023

    Streamlining Airflow Workflows with Enterprise Automation

    What was released?

    Airflow integrations are now available for AutoSys, Automic Automation, and dSeries. Apache Airflow is an open-source workflow management platform used to develop, schedule, and monitor batch-oriented processing. Airflow is also available through a managed offering from the major cloud vendors. Automating execution from within Airflow creates an additional island of automation for IT Operations to manage.

    Automation by Broadcom allows you to easily integrate directed acyclic graphs (DAGs) - a collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies - with your existing enterprise workload automation solution. Not only does Automation by Broadcom integrate with the open source Apache Airflow, we also integrate with the managed offerings from cloud vendors including Google Cloud Composer, Amazon Managed Workflows for Apache Airflow (MWAA), and Azure Data Factory Managed Airflow.

    What do the integrations do?

    They easily integrate Airflow DAGs with your existing Broadcom enterprise workload automation. You instantly inherit the advanced capabilities of your enterprise solution to deliver your digital transformation quicker and more successfully.

    By extending your enterprise automation to Airflow you maintain end-to-end visibility, regaining centralized command and control, alerting, SLA management, reporting, and auditing.

    With the Airflow integration, you can:

    • Schedule execution of Airflow DAGs
    • Monitor to completion of Airflow DAGs
    • Retrieve Airflow DAG logs for problem diagnosis
    • Restart Airflow DAG processing
    • Alert on the failure of Airflow DAG processing
    • Create dependencies between Airflow DAG executions
    • Incorporate Airflow DAG processing in existing enterprise automation workflows including dependency enforcement

    What are the scheduling limitations of Airflow?

    While it may be easy for developers to get started with Airflow, many challenges can arise once applications are handed over to production. This is particularly true for teams in large enterprises.

    When running applications in production environments, teams need to ensure adherence to standards and policies and support ongoing audits. Airflow presents many limitations that make these day-to-day operations difficult.

    Limited scheduling options

    Airflow’s scheduling capabilities are too limited to address the business needs of most enterprises. To schedule DAGs in Airflow, teams are limited to date/time criteria. Most business applications require much more scheduling sophistication.

    Isolated, complex operation

    With Airflow, there is no visibility into corporate-level calendars. That means something that should be simple, such as preventing workloads from running on holidays, is not very simple. This type of logic is managed at the individual DAG level. Any time calendaring changes need to be made, teams would need to make those changes individually for each DAG that was affected.

    Even simple changes to a DAG schedule can be overly complicated, potentially requiring either the renaming of the DAG or editing data in the database. Further, within most organizations, multiple development groups will employ Airflow. Each deployment will have its own user interface. Consequently, teams will need to use a distinct UI for each Airflow deployment, and for any other tools being employed.

    Limited reporting and visibility

    In Airflow, reporting and auditing are virtually non-existent. Any limited analytics that are captured and displayed are at the individual DAG level. Additionally, you can’t view a failed workflow and quickly determine what the business impact is for any downstream work.

    Limited capabilities and controls

    When using Airflow, you can’t easily have dependencies between DAGs. Effectively, all dependencies need to be defined within a single DAG. With Airflow, you can’t easily kill a job, implement a one-time override, define priorities, or take advantage of features like virtual resources.

    What are the benefits of integrating Airflow with Broadcom’s Automation Solutions?

    Quite simply, Airflow lacks many of the standard, out-of-the-box capabilities Broadcom users have relied upon and grown accustomed to. Despite these limitations, development teams may still opt to use Airflow, and, indeed, various groups are already deeply invested in Airflow. That’s why the integration capabilities of Automation by Broadcom are so valuable.

    There are many benefits to extending enterprise automation to cloud processing. With Automation by Broadcom, you can effectively establish a “manager of managers,” employing a unified tool to manage several disparate cloud scheduling and workflow orchestration solutions and get through end-to-end business process visibility.

    Through these integrations, organizations get the best of both worlds. Development teams can use Airflow if they choose while operations teams get the power of enterprise-grade schedulers.

    Where can I learn more about Broadcom’s cloud integrations for workload automation?

    Broadcom’s Automation Marketplace makes it easy to browse and search for cloud integrations for Automation by Broadcom as they become available. There you can find information on all the Airflow integrations from Broadcom including the open-source Apache Airflow, Google Cloud Composer, Amazon MWAA, and Azure Data Factory Managed Airflow.

    Visit Automation Marketplace for details on integration features, links to extended TechDocs, and the option to download the software.

    Richard Kao

    In his 30-plus year career at Broadcom, Rich has helped design, build, and support workload automation solutions. As a Distinguished Solution Architect, he has spent the last 16 years focused on helping customers solve complex business issues through automation.

    Other posts you might be interested in

    Explore the Catalog
    October 26, 2023

    Cloud Automation Integrations for Azure from Automation by Broadcom

    Read More
    October 13, 2023

    Streamlining Kubernetes Operations with Enterprise Workload Automation

    Read More
    October 2, 2023

    AutoSys Sees Continued Innovation, Expanded Multi-Cloud Support

    Read More
    September 29, 2023

    Announcing the General Availability of AAI 6.5

    Read More
    September 25, 2023

    Change Modeling: 5 Must-Have Capabilities for Your Workload Automation Solution

    Read More
    September 20, 2023

    Streamlining Azure Data Factory Workflows with Enterprise Workload Automation

    Read More
    August 31, 2023

    Survey Finds Using Multiple Cloud Automation Solutions Reduces Compliance and Creates Operational Challenges Leading to Consolidation Trend

    Read More
    August 24, 2023

    Why Enterprise Automation Is So Important for Compliance

    Read More
    July 26, 2023

    Streamlining Databricks Workflows with Enterprise Workload Automation

    Read More