AWS Integrations for Automic, AutoSys, and dSeries

While organizations are seeking efficiencies by using built-in workload automation from AWS, cloud-native tools create islands of automation leaving IT Ops blind to dependencies between jobs and the impact that delays have on SLAs. AWS integrations for Automic Automation, AutoSys, and dSeries provide the ability to leverage advanced scheduling for AWS workloads as well as provide end-to-end visibility for the workload regardless of whether it is on premises or in the cloud.

Amazon Web Services (AWS) was launched back in 2006, and it has persistently remained at the top of the market since that time. In 2022, annual revenues reached $80 billion. In the third quarter of 2023, AWS had 32% of the global cloud infrastructure services market and generated $23 billion in revenue.

Today, there are more than 1 million active AWS users in 190 countries. Over the years, the breadth of services has also grown dramatically. Launched as a virtual server and storage service, AWS now has more than 70 offerings, including networking, database, and analytics.

Limitations of managing workload automation in AWS

By moving workloads and automation to AWS and other cloud-based services, organizations are seeing a number of benefits. However, significant challenges remain.

Lack of business process visibility

These days, organizations will typically have business services that rely upon workloads running in AWS and other environments. While it is possible to use AWS-based tools to manage their workloads in AWS, these tools lack some key capabilities. For example, these offerings typically don’t provide complete visibility of the end-to-end automated process, they don’t track dependencies between jobs, and they don’t provide insights into potential SLA breaches.

Limited scheduling capabilities

When running automated jobs in AWS environments, teams can use AWS-based scheduling capabilities. The problem is that these technologies can’t intelligently accommodate dependencies—and in most enterprises, these workflows typically have multiple upstream and downstream dependencies. The only option is to create hard-wired time delays, that is, scheduling subsequent tasks to start at a time after which prior tasks are expected to have been completed.

Following are just a few of the challenges these approaches present:

• Reliability and data quality issues. Relying on these hard-wired schedules makes processing environments highly brittle. If one task takes longer than the forced time delay established, a subsequent task will either fail or move forward with incorrect or incomplete data. This means the output will be suboptimal or unusable. These issues get magnified in many large-scale environments, where dozens of data sources may be used. If one source doesn’t come across in time, many downstream workflows may encounter failures.
• Connectivity-related failures. In AWS environments, services rely upon a number of dispersed, distributed networks. At any given moment, automation jobs can fail due to intermittent connectivity issues. For example, a call to an API may temporarily return an error message. When this type of failure occurs, scheduled jobs will fail, often creating cascading issues for subsequent downstream jobs.
• High costs and poor resource utilization. To try and prevent some of the issues outlined above, teams can try to add buffers, that is, delaying the start time of subsequent jobs to accommodate potential delays. While this approach can help reduce failures, it isn’t foolproof. Further, this approach requires AWS instances to be kept idling, often unnecessarily. This can be very costly, potentially leading an organization to incur additional expenses of thousands of dollars a week.
• Labor-intensive follow-up and mitigation. If administrators find out about a failure while a workstream is being processed, they will have to disable the schedule, potentially in multiple products, and manually troubleshoot and address any issues that have arisen.

The solution: Cloud integrations from Automation by Broadcom

With cloud integrations for Automic Automation, AutoSys and dSeries, teams can harness advanced scheduling and manage dependencies across pipelines, integrations, applications, and processes. These solutions deliver end-to-end visibility across all automation workloads, including those running in on-premises deployments and various cloud environments, including AWS.

Today, Automic Automation, AutoSys, and dSeries offer integrations with these AWS solutions:

• AWS Batch. AWS Batch is a compute service that plans, schedules, and runs your containerized batch or machine-learning workloads across the full range of AWS compute offerings. With AWS Batch integrations, Automaton by Broadcom provides easy automation for data analytics workloads and scheduling for high-throughput screens. Teams can simply define AWS Batch workloads and allow their Automation by Broadcom solution to manage the scheduling of these workloads and at the same time, leverage the advanced workload automation capabilities of their enterprise automation solution.
  
• AWS Lambda. With AWS Lambda, teams can run code without provisioning or managing servers. Users can set up code so that it can be automatically triggered by AWS services like S3 or by calls from any web or mobile app. By leveraging Automation by Broadcom integrations, customers can centrally orchestrate these Lambda workloads, along with their other automation jobs.
• Amazon Simple Storage Service (Amazon S3). S3 provides object storage that enables customers to store and retrieve any amount of data from anywhere on the web. Our integrations enable customers to centrally manage S3 buckets. Teams can orchestrate a range of tasks, including copying, moving, deleting, and synchronizing files between other S3 buckets or local file systems.
• Amazon EventBridge. EventBridge is a scalable, serverless, fully managed event bus. With the solution, teams can use events to integrate applications. These events can come from different sources, such as applications, AWS services, and third-party SaaS applications. Automation by Broadcom makes it easy to integrate EventBridge events with existing enterprise workload automation and other cloud-native activities. By extending existing enterprise automation to EventBridge, teams can establish end-to-end visibility and centralized command and control.
• AWS Glue. Glue offers users a console for defining and orchestrating extract, transform, and load (ETL) workflows. While Glue offers a scalable approach for moving and transforming data, this tool effectively creates another island of automation that IT operations has to manage. Automation by Broadcom makes it easy to integrate Glue workflows with your existing enterprise workload automation solution. Users simply define their processes in Glue and then enable their Broadcom enterprise workload automation solution to manage the scheduling of these workflows.
• AWS Step Functions. Step Functions is a serverless orchestration service that integrates with various AWS services and is used for building workflows. When employing Step Functions to automate various AWS services, teams are effectively creating an isolated automation function that IT operations teams have to manage separately. With AutoSys, teams can easily integrate Step Functions into their enterprise workload approaches. Simply build your workflows within Step Functions and use AutoSys to manage the scheduling of these processes. AWS Step Functions are available for AutoSys and dSeries customers with an integration for Automic coming in the future.
• MWAA (Managed Workflows for Apache Airflow). Amazon MWAA is a managed workflow automation service built on Apache Airflow, used to develop, schedule, and monitor batch-oriented processing. It allows teams to incorporate DAG processing in existing enterprise automation workflows including dependency enforcement.

Benefits of Enterprise Automation by Broadcom

There are many benefits to extending enterprise automation to cloud processing. With Automation by Broadcom, you can effectively establish a “manager of managers,” employing a unified tool to govern several disparate cloud scheduling and workflow orchestration solutions. In this way, you can gain end-to-end business process visibility across all your cloud and on-premises workloads.

By extending your existing enterprise automation to AWS-based business processing, you maintain end-to-end visibility and you regain centralized command and control. The solution gives you unified automation observability of your cloud processing. Automation by Broadcom enables you to harness the power of predictive analytics with smart alerting, as well as advanced SLA management, reporting, and audit capabilities.

Learn more about Broadcom’s cloud integrations

Broadcom’s Automation Marketplace makes it easy to browse and search for cloud integrations available for Automation by Broadcom. The site features information on all Broadcom cloud integrations, including those for AWS as well as Azure, Cloud Foundry, Databricks, Google Cloud Platform, and many more.

Automation by Broadcom continues to rapidly deliver cloud processing integrations to provide our customers with the workload automation capabilities they expect from their enterprise solution. Visit the Automation Marketplace to get access to integration details, technical documents, and software downloads.