Confluent makes Apache Kafka cloud-native

Increasingly, the future of computing is in server-less, "cloud native" architectures, where Platform as a Service (PaaS) gives way to consumption-based pricing and thinking less about discrete infrastructure. This is especially true in the data world, and with good reason: setup and maintenance of the infrastructure -- and the always-on hourly fees that go with it -- is a headache. This is especially true for databases like Apache Cassandra and HBase and data collection/analytics technologies like Hadoop, Spark and Kafka.

Also read: Confluent's Kafka distro adds dev, management and hybrid cloud capabilities

For the last of these, Apache Kafka, such no-headache setup has been elusive. But now, Confluent, the major commercial entity behind the open source event streaming platform, founded by its creators, is changing that. Specifically, Confluent Cloud, the company's heretofore PaaS offering for Kafka, now offers a consumption-based pricing model, with no commitment on minimums. The news was announced yesterday at in London at Kafka Summit, which Confluent sponsors. That same day, Gaetan Castelein, Confluent's VP of Product Marketing and Justin Dorff, the company's Director of Communications, briefed ZDNet on the details.

Spec sheet

Confluent explained the new service will allow customers to scale usage of their server-less infrastructure, up and down, between zero to 100 MB/s (that's megabytes, not megabits), with billing based on volume of data ingress and egress, as well as volume of data persisted. Throughput rates higher than 100 MB/s are available on Confluent Cloud Enterprise, but only with the conventional provisioned infrastructure model -- and the two models cannot be combined for a single Kafka topic.

According to the company, Confluent Cloud is the first cloud-native Kafka service. Of course, each of the major cloud vendors offer their own streaming data services and one of them (Microsoft's Azure Event Hubs service) even offers Kafka API compatibility. Amazon Web Services (AWS) also offers its own Managed Streaming for Kafka (MSK -- in preview), though that is a PaaS offering, wherein users provision a cluster and pay based on virtual machine types and cluster up-time, rather than consumption.

Companion services

The Confluent Cloud consumption-based service is available on AWS and Google Cloud Platform, and includes fully-managed versions of Confluent Schema Registry, the KSQL query language (which would otherwise require additional infrastructure of its own) as an alternate API and, in the AWS case, the Kafka Connect S3 connector, which allows streamed data to be delivered to cloud object storage.

Also read: Confluent, Fivetran announce deep Google Cloud integration
Also read: Arcadia Instant for KSQL: Streaming data analytics for the masses?

The fully-managed Schema Registry, KSQL and Kafka Connect S3 connector services are being launched in public preview.

Et tu, public cloud provider?

This iteration of Confluent Cloud will lower cost-model barriers to entry in much the same way that KSQL ushered in skill set barriers that were lower. The ball is now simultaneously in Microsoft, AWS and Google's courts to make their own Kafka services equally consumable, or perhaps to partner more closely with Confluent to offer first party "endorsement" of Confluent Cloud.