FoundationDB: A new take on an established data structure

It appears that distributed, SQL and NoSQL database engines are all the rage for extreme transaction processing or Big Data applications. Some of these systems trade distributed performance for transactional integrity. Some will tell the program that data has been stored and is safe and it really is living somewhere in some system's memory and hasn't actually been safely stored on a disk somewhere.

FoundationDB's Ori Herrnstadt dropped by to explain his company's offering and why "eventual consistency" can be challenging.

During the discussion of the company's "key value store," I found myself reminded of the Pick and MUMPS databases I worked with long ago. There it is. I said it. I was a MUMPS practitioner and database engineer.

Eventual Consistency

Some distributed databases, regardless of whether they support the SQL query language or something else (NoSQL), replicate data among a number of systems and tell programs that the data has safely been stored even though it really hasn't really gotten to a reliable storage media. The suppliers offering these systems point out that is very, very unlikely that all of the systems holding the data would become unavailable at the same moment.

Data, they would point out is "safe enough."

FoundationDB believes that "safe enough" isn't really safe. It is quite possible that multiple systems in a data center or, perhaps, the whole data center itself to become unavailable. Data could be lost during one of these rare outages.

Eventual consistency isn't the same as data consistency, Hernstadt pointed out.

FoundationDB's approach

FoundationDB has built a database engine based upon the concept of a key-value store, which is a database that is designed to use the data itself as its own index. Developers can access this data store using a set of APIs or through language processors that take traditional query languages and break them down for processing by the key value store.

The key value store supports sharing, hierarchical storage, multi-map storage and quite a number of advanced database concepts.

An important point is that FoundationDB has developed a way to make sure that transactions made using its key value store are ACID compliant. Since these transactions are operating in a distributed environment, they are also very fast.

One of the company's messages is that its database makes it possible for many different database engines to be replaced with a single database because all of the functions they offer are available in FoundationDB.

Snapshot analysis

It was refreshing to talk about a database that was built upon the same ideas as Pick and MUMPS (now called M). The database is built on a highly compact B-tree architecture that uses the data as part of the index. Data is not constrained into specific types. A node in the database can be a string, a number, a pointer to lower level data or some combination of data and a pointer. 

This data can be structured in a number of ways that makes it possible to implement quite a number of different storage mechanisms.

This data structure reminds me of my work with Pick an MUMPS databases long ago. FoundationDB has taken the concept of a B-tree database, the foundation of both the Pick and the MUMPS database and has run quite a way with it.

In FoundationDB, that database engine is part of a highly distributed, highly reliable, ACID compliant database engine. The engine is constructed of three layers — a storage layer that implements the actual data storage and retrieval capability, a distributed processing layer that allows database components to reside on a cluster of systems and automatically place data where it should be placed for best performance, and a language processing layer that allows database access using a number of different queries. At this time, SQL is available. Other query language support can be expected in the future.

I was highly impressed with the cleverness of FoundationDB's approach and expect to hear big things from them about support of both Big Data and extreme transactional applications in the future. If you are seeking a way to consolidate a number of different database engines to reduce staff and licensing costs, FoundationDB may just be the right answer.

If you are looking for tools to implement Big Data applications, FoundationDB may also be the answer you've been looking for.