Full SQL on Hadoop? Splice Machine opens up its database for trials
Splice Machine says the launch this week of its public beta opens up the Hadoop stack for the first time to a full-featured SQL database, capable of running transactions and analytics simultaneously.
The company, which has been working with 15 charter customers and raised $15m funding in February, has made its eponymous product available as a free download to encourage testing and development.
"It's radically different to anything else that's out there because it's the first true ANSI SQL transactional database on the Hadoop stack. It can power concurrent applications. People can read and write from the database at the same time," Splice Machine CEO and co-founder Monte Zweben said.
"This is not just data science anymore for Hadoop, where a batch of data is loaded up into the Hadoop file system, you run some analytics on it using MapReduce or even a SQL layer, and then dump the results back into a report.
"This is about real-time, concurrent applications, and this has not been feasible until now."
Splice Machine has taken the Apache Derby Java relational database and removed its storage layer, replacing it with the Apache HBase NoSQL database. Then the company modified the planner, optimiser and executor inside Derby to take advantage of HBase's distributed architecture.
"Now what happens is that Derby, like its original version, compiles a SQL plan out into byte code — a very efficient representation of a SQL execution — and we can distribute that out to the HBase nodes, so the computation can take place in parallel and close to where the data is stored for the maximal efficiency," Zweben said.
"Then we splice the results back together again — hence the name of the company."
According to Zweben, the skills required to use the Splice Machine software are a knowledge of SQL and some familiarity with the Hadoop file system, to be able to configure and install it.
"But they do not need to be Java programmers and they do not need to know MapReduce," he said.
"The significance is that heretofore being able to power real-time applications on the proven Hadoop stack was limited only to very low-level, key-value storage systems like HBase.
"The vast community of application developers and IT people really could not take advantage of it because they would have to be able to program in Java."
Zweben said Splice Machine is not offering all its code back to the Hadoop and HBase projects but it is a contributor.
"We fix bugs and contribute to places where it makes sense for the whole community back into HBase and Hadoop as good participants in the community. But we also have some proprietary software as well — kind of like what everyone does," he said.
"They keep some IP proprietary for the shareholders of the company but they contribute to the open-source community openly and we are just like that."
Zweben said the database market is currently confusing because of the variety of options available but for those whose databases are under pressure the first choice is whether to scale up, with expensive proprietary hardware, or scale out on cheaper commodity clusters.
Even if they choose scale-out, people then face three options.
"The first is in my opinion a very poor choice and that's NoSQL. NoSQL definitely has the scale-out feature in its architectures but unfortunately you throw the baby out with the bath water," Zweben said.
"All the services that SQL provides need to be written now at the application level by the developers, which is costly and error prone, whether that is aggregation or joins or transactions. All these things, plus all the tools around SQL, have to be rewritten and that's a mistake.
"NoSQL may be great for simple web pages but if you're doing a real app you need an SQL database."
So if the choice is SQL, the options become whether to scale out on a proprietary architecture or on Hadoop.
Tech Pro Research
"People around the world developing on Hadoop and HBase — two scale-out and open-source technologies — are contributing so much IP and technology in terms of both the architectures of Hadoop and HBase, plus all the systems around Hadoop and its ecosystem," Zweben said.
"Our competitors in the NewSQL space, who are essentially scale-out vendors, they have to write all that code themselves. They have to write a distributed file system. They have to write a fast key value store. They have to build all the other tools."
Zweben said new versions and new functionality for Splice Machine will be announced this year, including work on architectural extensions such as in-memory computation.
The ACID-compliant product is available on a freemium basis for test developments and experimentation. Used in production environments, Splice Machine has a list price annual licence fee of $5,000 per node.