Get the Best of Both Worlds: How Automation by Broadcom Cloud Integrations Boost Google Cloud Workflows

Over recent years, the cloud services market has continued to see rapid growth, and Google Cloud has had a significant role to play in that expansion. The service was launched in 2008 and by 2022 was generating $26.28 billion in annual revenue. Customers rely upon Google Cloud to build, deploy, and scale applications, websites, and services. Google Cloud hosts more than 1.42 million websites.

Since the launch of the service, Google has continued to expand the range of offerings available. Currently, the Google Cloud features more than 150 products, including offerings in such categories as AI and machine learning, computing, storage, databases, data analytics, networking, and more.

Limitations of managing workload automation in Google Cloud

While teams are gaining efficiencies and other benefits by moving workloads and automation to Google Cloud, significant challenges remain.

Lack of business process visibility

These days, organizations will typically have business services that rely upon workloads running in Google Cloud and other cloud environments. For example, one survey found that 85% of organizations deploy applications in two or more IaaS provider environments. While it is possible to use Google Cloud tools to manage Google Cloud-based workloads, these tools don’t provide complete visibility of the end-to-end automated process, they don’t track dependencies between jobs, and they don’t provide insights into potential SLA breaches.

Limited scheduling capabilities

In Google Cloud environments, teams can employ basic, time-based schedulers. However, these technologies can’t intelligently accommodate dependencies—and in most environments, these workflows typically have multiple upstream and downstream dependencies. Given these limitations, the only option is to create hard-wired time delays, that is, scheduling subsequent tasks to start at a time after which prior tasks are expected to have been completed.

Here are some of the obstacles presented by these approaches:

• Data quality and reliability issues. When teams rely on hard-wired schedules, the result is that if one task takes longer than the forced time delay established, a subsequent task will kick off, typically with old, inaccurate, or incomplete data. This means the job sequence may return suboptimal or unusable data or fail completely. The larger and more complex the environment, the more these issues intensify. For example, it’s common for teams to have jobstreams that rely on dozens of data sources. If just one source doesn’t come across in time, workflows may experience cascading failures.
• Network-related failures. When running Google Cloud, users are highly reliant upon a range of dispersed, distributed networks. At any given moment, automation jobs can fail, simply due to temporary connectivity problems. For example, a call to an API may return an error message, but moments later, the same call may complete successfully. Even when these brief outages occur, jobs running on hard-wired schedules will fail, introducing issues for subsequent downstream jobs.
• High costs and inefficiency. To prevent these issues, operators can opt to add buffers, that is, delaying the start time of subsequent jobs to accommodate potential delays. While this approach can help minimize failures, it doesn’t by any means eliminate them completely. Further, this approach results in the need to keep Google Cloud instances idling, often unnecessarily. This can be very costly. In large enterprise environments, costs associated with these idling resources can exceed tens of thousands of dollars a month.
• Labor-intensive remediation. If a failure is discovered while a workstream is underway, an administrator will likely be forced to take on a labor-intensive effort to investigate and remediate. They may have to disable the schedule, potentially in multiple products, and manually troubleshoot and address any issues.

The solution: Cloud integrations from Automation by Broadcom

Automation by Broadcom offers robust scheduling that enables users to manage dependencies across pipelines, integrations, applications, and processes. These solutions deliver end-to-end visibility across on-premises deployments and various cloud environments, including Google Cloud.

Today, Automic Automation, AutoSys, and dSeries offer integrations with these Google Cloud solutions:

Google Cloud Storage

Google Cloud Storage is an object storage service that enables customers to store, access, and manipulate data. This storage is used for a wide range of workflow processing use cases, so being able to manage these objects effectively is key to automation success. With our integrations, Broadcom makes it easy to integrate the monitoring and management of this cloud storage and other cloud-native activity with your existing enterprise workload automation solution.

Google Cloud BigQuery

Google Cloud BigQuery is a fully managed enterprise data warehouse. With BigQuery, teams can manage and analyze data using machine learning, geospatial analysis, and business intelligence. Within BigQuery, it is possible to automate job execution, however, this creates an additional island of automation for IT operations to manage. With Automation by Broadcom, you can define your data transfers and schedule queries within BigQuery and use your enterprise workload automation solution to manage the scheduling of these workflows.

Google Cloud Composer

Google Cloud Composer is a fully managed workflow orchestration service. Teams can use the service to create, schedule, monitor, and manage workflows, including those that span across clouds and on-premises data centers. However, when teams use Cloud Composer, they are effectively creating a new isolated automation environment that IT operations has to continue to track and support. Our integrations enable teams to easily integrate Airflow-directed acyclic graphs (DAGs) with their existing enterprise workload automation solution. With these capabilities, teams can maintain end-to-end visibility and regain centralized command and control.

Google Cloud Data Fusion

Google Cloud Data Fusion is a popular cloud-based service that enables customers to quickly build and manage data pipelines. The service features capabilities for data extract, transfer, and load (ETL) and data integration. Now, you can use Automation by Broadcom to manage the scheduling of your Data Fusion pipelines.

Google Cloud Dataflow

Google Cloud Dataflow is a serverless data processing service for stream and batch data. As a customer’s data volumes grow, the service provides automated infrastructure provisioning and scaling. While Dataflow enables teams to automate job execution, this creates a separate automation service that IT operations teams have to manage, which can introduce complexity and a lack of visibility into potential issues. With our integrations, you can use Automation by Broadcom to centrally manage Dataflow workloads, as well as those of your other services and environments. Note: This integration is currently only available for AutoSys and dSeries.  Integration with Automic will be available in the future.

Benefits of Automation by Broadcom

There are many benefits to extending Automation by Broadcom capabilities into Google Cloud. With this solution, you can establish a unified tool to govern your disparate cloud scheduling and workflow orchestration tools. The solution gives you unified automation observability of all your cloud and on-premises workloads.

By extending your existing enterprise automation to Google Cloud-based business processing, you maintain end-to-end visibility and you regain centralized command and control. With Automation by Broadcom, you can leverage the power of predictive analytics with smart alerting, as well as advanced SLA management, reporting, and auditing capabilities.

Learn more about Broadcom’s cloud integrations

Broadcom’s Automation Marketplace makes it easy to browse and search for cloud integrations available for Automation by Broadcom. The site features information on all Broadcom cloud integrations, including those for Google Cloud, as well as AWS, Azure, Cloud Foundry, Databricks, and many more.

Automation by Broadcom continues to rapidly deliver cloud processing integrations to provide our customers with the workload automation capabilities they expect from their enterprise solution. Visit the Automation Marketplace to get access to integration details, technical documents, and software downloads.