Hortonworks ups its Kafka Game

For Hortonworks' largest customers, streaming is becoming part of the mix. Over the past quarter, 10 of Hortonworks' 17 million-dollar plus deals included Hortonworks Data Flow (HDF), the platform that curates and routes data in motion to Hadoop clusters. And as much of this data flows in and out of the cloud, Hortonworks DataPlane Service (DPS) was launched to provide a window for governing data flowing across hybrid or multi-cloud environments.

That's the context for today's announcement of Streams Messaging Manager (SMM), a new DPS module that adds more visibility to Kafka message queues. The new offering visualizes what's going on in Kafka clusters, tracking traffic flow and bottlenecks. With integration to Apache Atlas and REST APIs that expose monitoring and management capabilities to third parties, SMM provides several paths by which Kafka traffic can be covered through governance umbrellas. For instance, by integrating with Atlas, lineage is now supported for Kafka down to the topic level.

The obvious comparison is with Confluent Enterprise, the original product in this space that provides a wide range of capabilities, from schema registry to cluster rebalancing, replication, and its own control center pane of glass. By comparison, SMM leverages the capabilities of the Hortonworks Data Platform for functions such as cluster management. But its differentiation is in the visualization; with SMM, you can see Kafka Producers (sources that send data) without requiring the need to build a client (Interceptor) to identify which producers are sending which messages.

SMM is not a standalone offering, but the newest tool in the DPS portfolio. To recap, DPS operates as a catalog of catalogs for registering data services that would otherwise be difficult to track in hybrid environments. Consider it a skeleton atop which specific tools are plugged in to provide visibility into different functions of the Hortonworks platform. SMM follows Data Analytics Studio, for exploring Hive metadata; and Data Steward Studio, for associating clusters with specific Hadoop NameNodes.

Along with the SMM release is a dot release refresh of Hortonworks HDF. The new version 3.2 adds refinements, such as Kerberos keytab isolation for improving control over the multi-tenant environments that are commonplace in the cloud and improved streaming performance through support of Hive 3.0.

The emergence of IoT has pushed streaming to the front burner. Capture and analysis of IoT data has been one of the prime use cases that have triggered HDF take-up. But other use cases, such as managing data movement between on-premise and cloud data centers and real-time ingest of Customer 360 data demonstrate that IoT is not the only game in town for streaming.