Speedb is a drop-in replacement for RocksDB that wants to take the embedded key-value store world by storm
Key-value stores are the simplest, most fundamental data storage systems. Rather than providing elaborate query languages, most of them provide simple interfaces to store and retrieve data.
Conceptually, any type of data management system or database can be built on top of a key-value store. More elaborate data structures and query languages can be, and sometimes are, built on key-value stores.
In that scenario, database builders usually either implement their own key-value store solution or look for an embedded system. That is a key-value store that comes as a library that other systems can use.
It could be argued then that embedded key-value stores are the unsung storage heroes and frequently for databases. Embedded key-value stores come with the additional baggage of embedded systems: they share resources with the system embedding them, which means efficiency is key. In the words of Adi Gelvan, Speedb co-founder and CEO:
"If it's outside the system, then it's someone else's problem, and you can actually get it as a service. But when you're talking about an embedded library, it has a direct impact on your application and resource utilization".
Speedb is a new embedded key-value store that offers 100% compatibility with RocksDB, the popular open source embedded key-value store while boasting a significant boost in performance.
Today Speedb is announcing its official launch, its seed round funding, as well as a partnership with Redis. We caught up with Gelvan to learn more about Speedb.
A drop-in replacement for RocksDB
Gelvan met his co-founders while working at Infinidat, a big storage company, around 2011. But it was not until 2019 that the idea of Speedb was born. What sent them down the rabbit hole was what Gelvan called "metadata sprawl".
As he put it, metadata used to be an insignificant part of data, but now sometimes, it's bigger than the data itself. For one of their projects, Speedb founders were looking for a way to handle metadata very effectively without having to add more hardware to the storage system.
One option was to rewrite the whole data stack, which was not really an option. The second one was to look for available solutions to manage metadata effectively. They were looking for an embedded solution, so it did not take them long to arrive at RocksDB.
RocksDB is a Facebook open source project, started as a fork of Google's LevelDB, and it has a very large community and user base. There's just one problem: it could not scale to what was needed. After reaching 100GB, Gelvan and his co-founders started running into all sorts of issues.
They reached out to the RocksDB community and discovered that this was expected behavior. The solution that was offered was to shard RocksDB. That is, to break datasets into smaller pieces, and then assign each one to a node, and have every node run with its own storage engine.
Speedb's founders were not happy about that, as sharding introduces complexity and overhead. So they started thinking about ways to go around sharding by creating what Gelvan called a next-generation storage engine:
"We found out that by providing extra performance and scale using less resources, we can allow our customers to run much more data at a much higher speed. They can change as much as they want on the application side of things and not be really limited by the data side of things", Gelvan said.
In early 2020, Gelvan and his co-founders decided that they could re-implement log-structured merge (LSM) trees, the data structure at the core of RocksDB, in a way that is more efficient than what RocksDB offers. They also figured out that if they did it in a way that maintains compatibility with the RocksDB API, they would be able to address RocksDB users, and that all adds up to a sizeable market.
100x more data, 10x faster, 80% fewer resources
That's easier said than done, of course, but it looks like Speedb has delivered on that plan. About 18 months later, Speedb has a working product. In fact, one was selected to support the largest deployments on Redis on Flash, allowing the upcoming version to generate 2x more throughput while cutting latency by half.
As Gelvan mentioned, they were looking for early adopters, and Redis' offices is a couple of floors from where he works, so he started a conversation. Redis' people said they'd be happy to give Speedb a go if it was indeed a drop-in replacement for RocksDB. Gelvan said that it took Redis' people about 30 minutes to get Speedb up and running, run a few benchmarks, and call back Speedb to talk business.
The Speedb vs RocksDB story has some similarities to the ScyllaDB vs Cassandra story. A re-implementation of a popular open-source framework in a more efficient way while maintaining compatibility with the original. Gelvan noted that while many optimizations for RocksDB have been proposed, what they did was to rethink the foundation of the LSM data structure from the ground up.
Speedb has filed for some patents around that as well. The claim is that Speedb's technology enables 100x more data, 10x faster, and uses 80% fewer resources than RocksDB. Thus, it allows for lower CPU and memory consumption even as the data scales.
As opposed to RocksDB, Speedb is not open source, however. The idea seems to be that since RocksDB has such a wide user base, and Speedb offers such a performance boost, there will be a significant part of that user base that can be converted to Speedb paying customers.
Speedb has raised an initial $4M seed funding round from Hyperwise Ventures. However, Gelvan said they are looking to raise more capital. The obvious goal would be to expand the current team of 15 people, mostly geared towards research and development, with people who can help going to market.
However, that's not all Speedb has in store for the future. Gelvan also mentioned adding new features while potentially open sourcing parts of the product:
"We have lots of room to put other functionality that will actually help the application above to look inside the data and do smarter things with the data. And we are aware of the open source community, and this is something we are definitely considering. I would say stay tuned for the future", said Gelvan.
The company is also onboarding a number of clients and seems to be set for growth.