Google rolls out Cloud Composer for data orchestration
Video: Google offers cloud credits scheme to help academic researchers
Featured
Google Cloud on Tuesday is rolling out a new data orchestration tool called Cloud Composer. It's based on Apache Airflow, an open-source project focused on workflows, created Airbnb.
In recent years, Google has made a concerted effort to win corporate clients and take on Amazon Web Services in the public cloud. It's done so in part by offering a range of competitive cloud services and platform tools. However, that pitch to corporate clients "kind of falls apart if you don't have a great way to orchestrate and tie these tools together," James Malone, a Google Cloud product manager, said to ZDNet.
Cloud Composer, which is launching Tuesday in open beta, aims to solve that problem by giving customers an easy way to use all of Google Cloud's tools.
Google chose to build a managed Apache Airflow service for a variety of reasons. For one thing, all Airflow scripts are based on Python, which is highly approachable. "Anyone can spend an hour or two and become an Airflow developer," Malone said.
Read also: What's the best cloud storage for you? | Top cloud providers 2018: How AWS, Microsoft, Google, IBM, Oracle, Alibaba stack up | We found 22 cloud services your business definitely needs to try | Everything you need to know about the cloud, explained
Additionally, Google wanted to base the product on an open source project. Airflow has an active developer community, and a large library of connectors (called "operators") already exists for Google Cloud and other clouds. Airflow allows for orchestration of hybrid cloud environments, multi-cloud environments, and effectively all use cases.
The initial beta release of Cloud Composer includes features for easy and effective Airflow deployments, including client tooling including the Google Developer Console and Cloud SDK, identity access management, and stackdriver logging and monitoring.
Currently, Cloud Composer and Airflow have support for Google BigQuery, Cloud Dataflow, Cloud Dataproc, Cloud Datastore, Cloud Storage, and Cloud Pub/Sub. So, a customer could use it, for example, to run a query with BigQuery and then export the results to Cloud Storage.
Cloud Composer has been in private alpha for about eight months with a few hundred customers ranging from small educational institutions to very large enterprise institutions in finance and e-commerce. The large alpha, Malone said, was a reflection of the broad applicability of the orchestration tool.
Meanwhile, intent on being an active member of the Apache Airflow community, the Cloud Composer team has already submitted a over 20 pull requests to the Airflow project.
This article was corrected to note that Cloud Composer is in open beta.