Time and again, I am asked the same question: Is there still a role for AutoSys in today’s cloud-first world? The unequivocal answer is "yes." Here’s why.
Why is AutoSys a critical part of your cloud automation strategy?
Sometimes you need to look back to see the way forward. You have been using an enterprise-class workload automation tool for many years. It has enabled you to successfully deliver excellent service to the business and control all the workload (jobs) that need to be executed for the business to function.
By having everything under control in one place, you can deliver the service at minimal cost. After all, managing from many disparate places is hard and expensive. You can also deliver a better service because you are able to connect the processing (jobs) from different applications and assemble them into the business process. For example, meter to cash, replenishment, or bank reconciliation. You can also be proactive in improving these services as you better understand how all these jobs relate to the supported business.
Without enterprise automation, each application or machine is its own island of automation, running in isolation from anything else that drives the business. We can create semaphores or messages between these systems to connect them, but they are fragile, easy to break, and almost impossible to manage. Many years ago, Gartner, the analyst firm, coined these as “Islands of Automation.”
How does cloud automation change what I am running?
When you started, all the jobs you needed to run were on-premises. They may have been hosted in a data center, but you still had control over everything to run those jobs. Having agents on each of these machines made control over the jobs easy.
Then the cloud arrived. Not the cloud version you use to rent infrastructure, but the cloud where the application is hosted, managed, and run for you outside of infrastructure you control. Often this is called either software as a service (SaaS) or platform as a service (PaaS).
Many of us have moved our data warehousing from on-premises installation to cloud services, and others now consume their entire ERP system from a cloud vendor. Nearly every application we have on our on-premises environment exists as a managed service somewhere in a cloud.
We cannot install an agent on these services, so the problem we fixed decades ago – the Islands of Automation – resurfaces. Each of the cloud services we consume provides a way to automate jobs’ execution within their service, and connecting them together is a very manual process or not even possible with the cloud-native tools. So, Operations again has to monitor jobs across different locations, alerts are going to various places, if at all. But these jobs still have connected dependencies, in order for the business process to be successfully completed.
Gartner reported on this more recently in the Market Guide for Service Orchestration and Automation Platforms, or SOAPs for short.
How do I use AutoSys across these cloud applications?
All of these cloud services publish how to talk to them; they typically use RESTful web service calls, CLI or SDKs for various programming languages. Some still rely on the SOAP-based web service framework. AutoSys embeds the ability to integrate all types of these requests into your jobs. Why? because the complexity for cloud means you often need to make multiple calls to run a job (request it, iteratively monitor the job, retrieve the outputs of the job).
If you are working with one of the big cloud providers' services, they have a simple to use CLI like aws from AWS, az from Azure, or gcloud from GCP. For simple types of requests, that may be all you need to use. If you are comfortable with some of the major scripting languages, using the SDKs that these provide, you can easily create some really powerful what I call utility scripts. For example, starting and stopping EC2 images. I used this as a way to start learning Python and was pleasantly surprised to find lots of example code on the cloud websites in addition to all the outside examples. I created one to start and monitor an AWS Step Functions flow and even added a signal handler to be able to kill not only the script but to issue the request in Step Functions to stop the flow as well.
If you like using web services, you can build the jobs and flows you need to make calls to these services. You can pass the results and extract information to pass on to subsequent jobs all within the product. Often the first call is for login and returns a token for subsequent actions. That token can be passed to subsequent jobs automatically within the product.
So, it is easy to incorporate the cloud-based processing you have today within AutoSys, maintain that single view on workload, and enforce the dependencies across your business processes. The bottom line? You can maintain an excellent service to the business at the right operational cost.
But you are also not alone; you do not have to build everything yourself each time you want to use it – although that is easy to do.
Cloud automation tools
As some of the ERP products like SAP started encouraging customers to move to their SaaS offering, we began creating plugin extensions that work with these new services to help keep all the work invested in the past still relevant. We repeatedly worked with customers on some of the newer cloud services as well, so instead of just repeating similar workflows each time, the team at Broadcom began building these supported extension plugins for them.
Today we have a broad range of cloud automation integrations for for a variety of providers including for ETL solutions like Databricks and Azure Data Factory, and workflow automation tools like Apache Airflow and Google Cloud Composer. We also have integrations for ERP SaaS applications and cloud storage solutions like Amazon S3. Broadcom continues to release new integrations. The Automation Marketplace provides a one-stop shop for easy download of integrations and access to related documentation.
The beauty of this type of integration is it incorporates the submission and tracking of the processing in the cloud service. This means there is only one job for you to schedule. It will also complete with the final status of the cloud processing once it has finished, so your next job can then start knowing that the cloud processing is finished.
AutoSys is relevant in the cloud world. AutoSys has underpinned your successful operations for many years. It continues to do this as you absorb applications in the cloud and as part of your broader cloud strategy.
AutoSys is critical to Operations in order to maintain one place to monitor and manage the jobs being executed. So you can ultimately deliver the business processes your company relies on. The alternative is a return to the fragile environment we had originally, resulting in more service delivery failures and higher cost of operation.
