Confluent's Kafka distro adds dev, management and hybrid cloud capabilities

At today's Kafka Summit event in New York City, Confluent, the company founded by Apache Kafka's creators, will announce a new version of its distribution: Confluent Platform 5.2. The new release offers numerous features for developers, administrators and any customer that wants to handle streaming data processing across cloud and on-premises infrastructure.

Also read: Confluent release adds enterprise, developer, IoT savvy to Apache Kafka

The many flavors of Kafka

There's little doubt that Apache Kafka has become a de facto standard in the world of streaming data processing. But the fact that it's a standard, and that the core technology is open source, makes things interesting for Confluent. People can use the Kafka open source bits, after all, or use the version included in their Hadoop/Spark distributions from providers like Cloudera. MapR has its own Kafka API-compatible component called MapR Event Store for Apache Kafka. In the cloud (beyond Confluent Cloud itself), there's a public preview of Amazon Managed Streaming for Kafka (MSK) and Kafka API compatibility on Microsoft's Azure Event Hubs service.

Also read: Kafka 0.9 and MapR Streams put streaming data in the spotlight

With all these Kafka options, it's clear Confluent needs to add a lot of value if it's going to convince customers to go with its product. And with Confluent Platform 5.2, the company makes a good showing here. Yes, the release is based on Kafka 2.2, the open source bits for which were released a little over a week ago, on March 22nd. But Confluent's putting a lot more in the box.

Programming language utopia

To begin with, Confluent is including the 1.0 release of the "librdkafka" client library, which provides support for C/C++, and is now at near feature parity with the Java client. Moreover, since Confluent's clients for Python, Go and .NET (C#) are based on librdkafka, those clients get the same feature parity boost. And since the .NET SDK is crafted for .NET Core, it will enable .NET developers to write Kafka applications for Linux and MacOS, as well as Windows.

Meanwhile, in support of developers on all platforms, KSQL, the SQL-like query language for Kafka, has been significantly enhanced, with support for CASE, BETWEEN, GROUP BY and other SQL commands and keywords. As a bonus, Confluent Control Center, the company's browser-based tool for managing Kafka infrastructure, has enhanced its KSQL GUI for queries and added the ability to create, edit and validate schemas (as shown in the screenshot at the top of this post), as well as make changes to compatibility policy.

Also read: Arcadia Instant for KSQL: Streaming data analytics for the masses?

Better manageability, generous developer licensing

Speaking of Control Center, it now allows you to configure Kafka "brokers," rather than just monitor ones created at the command line. And Control Center can now scale to 120,000 partition replicas (i.e. 40,000 partitions, typically), which Confluent says should be sufficient to handle most high-end enterprise implementations.

What's even better for developers is that for settings where a single-broker Kafka cluster is sufficient, such as development and test environments, Confluent now offers a free developer license to its full platform. This means that even premium features (including Confluent's connectors, Control Center and Replicator) that are not available in open source Kafka or via the Confluent Community License can be had free-of-charge in non-production environments.

One more thing

Finally, for Kafka implementations that require streaming data to be processed across hybrid environments (e.g. on-prem and public cloud), Confluent Replicator now meets the requirement, and allows schemas to migrate across environments.

There's more to Confluent Platform than what I've covered here. The company has all the details in a blog post, which should be available when this article goes live.

Cloud services: 24 lesser-known web services your business needs to try

Big Data

How to find out if you are involved in a data breach (and what to do next)

Fighting bias in AI starts with the data

Fair forecast? How 180 meteorologists are delivering 'good enough' weather data

Cancer therapies depend on dizzying amounts of data. Here's how it's sorted in the cloud

• How to find out if you are involved in a data breach (and what to do next)
• Fighting bias in AI starts with the data
• Fair forecast? How 180 meteorologists are delivering 'good enough' weather data
• Cancer therapies depend on dizzying amounts of data. Here's how it's sorted in the cloud