Announcing the Availability of the Airflow Connector for AAI with the 6.5.2 Release

We are excited to announce the general availability of Automation Analytics & Intelligence (AAI) 6.5.2. Besides other valuable features, this release includes a connector for Apache Airflow and its managed variants in the public cloud. The Airflow connector addresses the growing challenges of monitoring and SLA management faced by organizations with increasing Airflow adoption within Data Engineering and Data Analytics teams.

Following is an interview with AAI product manager, Jennifer Chisik covering what’s new with AAI in this release.

What is included with the Airflow Connector for AAI in the 6.5.2 release?

With the Airflow Connector, AAI provides unified automation observability across on-premises schedulers and now, Airflow’s cloud-native as well as on-premise deployments. The release provides an AAI connector for both Apache Airflow and Google Cloud Composer. MWAA will be available in a subsequent release.

Why did Broadcom add support for Airflow data in AAI?

As organizations advance in their digital transformation journey, it is clear that cloud-based automation is an essential requirement in fulfilling the potential that this transformation presents.  While Broadcom offers support for automating the market-leading cloud-based solutions, large organizations also have teams that are taking advantage of open-source or cloud-native automation solutions, particularly Airflow.  

While there are conveniences associated with the use of Airflow, one of the problems that arises is the lack of a centralized view of all of the enterprise's automated applications.  With AAI’s newly introduced support for Airflow, we can provide visibility, in a single pane of glass, for siloed Airflow applications as well as the traditional on-premise automation solutions.

What are the limitations of Airflow’s analytics capabilities?

Many Airflow implementations are actually on laptops because it is a developer tool. And, then it is up to IT Ops to manage without true visibility into how the workload is performing.

Airflow reporting and auditing capabilities are pretty limited. Any analytics that are captured and displayed are at the individual DAG level. You can’t view a failed workflow and quickly determine what the business impact is for any downstream work. You are essentially missing the end-to-end process view with alignment to the business SLAs.

In addition to centralized visibility, some of the key features that AAI offers on top of what Airflow provides natively include SLA management, adaptive predictive analytics including proactive alerting if it is determined that a critical SLA is in jeopardy of being missed, historical trend analysis and reporting, critical path analysis essential for optimizing workflows, and a host of other advanced analytics capabilities.

What is the benefit of connecting AAI directly to Airflow?

By connecting AAI directly to Airflow, you can instantly gain visibility into your existing DAGs and take advantage of all of the sophisticated analytics that AAI provides.  It allows you to have a centralized view of all of your automation processes in one single location.  This includes the Airflow instances that pop up throughout an organization as well as the more traditional on-premise automation solutions that exist within the organization.

Can customers still use AutoSys and Automic Automation to orchestrate their cloud-native workflow?

Absolutely and we actually recommend that they do exactly that.  AutoSys and Automic offer advanced scheduling options that don’t exist within Airflow. And, they have the added benefit of allowing you to orchestrate Airflow as a component that coexists with other automation solutions. By allowing AutoSys or Automic to orchestrate your Airflow DAGs, you gain all of the advanced command and control capabilities, security, and resilience offered by these mature enterprise solutions.  With AAI in the mix, you get unparalleled deeper visibility into DAGs and unified visibility and analytics across all your workload automation solutions.  

What other new features are available in AAI 6.5.2?

In addition to the support for Airflow, the 6.5.2 release offers a new advanced job search facility that allows you to search for individual jobs based on any combination of job properties.  For example, you might want to find all of your jobs that belong to a particular business application and run on a particular agent. This advanced search is very flexible and in addition to being used as part of our global job search facility, it is a key component to many of the future data insights we are developing.

What’s next for AAI?

AAI will continue to evolve and expand the variety of analytics that it provides to help you manage your increasingly complex automation environments. Keep an eye out for new data insights, dashboards, and reports as they roll out in quarterly releases starting in mid-2024.  

In addition, AAI will continue to keep up with evolving cloud-based automation solutions, adding support for tools such as AWS Batch and other solutions we find our customers are gravitating towards.

How can customers provide feedback?

We are always interested in feedback from customers. There are several ways to do this which include the AAI Community, through Broadcom Support, and contacting me directly at jennifer.chisik@broadcom.com.

For more on what’s available, read the AAI 6.5.2 Release Notes and watch the AAI for Airflow demonstration.

To find out the results of a recent survey on the challenges organizations face in workload automation and see a demo of AAI 6.5.2, register for the webinar: Trending Challenges in Observability for Workload Automation.

Visit AAI on Broadcom Software Academy to learn more about AAI capabilities through the blog and other content, short demos, and free training.