Why Mesosphere is tapping into Spark and Cassandra for Infinity big-data and IoT stack

By assembling a set of open-source components in the Infinity stack, Mesosphere is aiming to simplify life for developers and data scientists working on big-data and IoT apps.

After announcing general availability in June of its DCOS platform for pooling datacenter and cloud resources, Mesosphere today showed how the technology can be used with a new software stack for building back-end, big-data and internet-of-things services.

Developed with Cisco, the Mesosphere Infinity open-source stack demonstrated at MesosCon in Seattle consists of Mesos-based DCOS, fast-growing analytics cluster framework Apache Spark, NoSQL distributed database Cassandra, high-throughput distributed messaging system Apache Kafka, and middleware Apache Akka.

Mesosphere describes these open-source technologies as the emerging de facto standards for developers and data scientists working on real-time data pipelines for big data and IoT apps. By putting these components into an interoperable stack for clouds and datacenters, the company aims to simplify deployment and operations.

"Building data pipelines can be extremely complex and time consuming, affordable only for the largest companies with significant technology budgets and deep engineering teams," Mesosphere CEO and co-founder Florian Leibert said in a statement.

​Mesosphere launches its Mesos-based datacenter OS plus a free version on AWS

As well as general availability for the enterprise version of its Datacenter Operating System, Mesosphere has also taken the wraps off a free community edition.

Read More

"The product removes operational obstacles and makes this combination of technologies accessible to the broad enterprise computing market."

Among its roles in big data and IoT, Infinity is designed to be used specifically for anomaly detection, predictive analytics, and delivering personalised customer services.

San Francisco-based Mesosphere, which also has engineering staff in Hamburg, offers an enterprise edition of DCOS, or Datacenter Operating System, as well as a free community edition on Amazon Web Services.

DCOS is based on the Apache Mesos open-source distributed systems kernel, the co-creator of which is Mesosphere chief architect Benjamin Hindman. Major web firms already using Mesos or the Mesosphere software include Airbnb, eBay, HubSpot, Netflix, Twitter and Yelp.

As well as collaborating on the development of Infinity, networking giant Cisco has optimised its Cisco Intercloud Services for the Mesosphere stack and says it already has customers running it in production.

On top of the work with Cisco, Mesosphere has teamed up with some of the companies closely associated with the stack's components.

These partners include services firm and major Apache Kafka contributor Elodina, Typesafe - whose founders created Akka and the Scala programming language, in which Spark, Akka and Kafka are written - and Confluent, which was founded by the creators of Kafka.

"These companies are experts in specific services and have been collaborating with Mesosphere to provide the underlying technology and will be assisting in the ongoing innovation, maintenance and support of these services," Mesosphere said.

Chip giant Intel has also been involved in the project, which it has validated as "suitable at scale on Intel hardware for the most demanding production deployment requirements".

More on Mesos and Mesosphere