Inside Microsoft's Cosmos DB

At Microsoft's Build conference in Seattle last week, the company introduced Cosmos DB, a new Azure cloud database as a service, superseding DocumentDB, the company's previous NoSQL offering. While ZDNet's Mary Jo Foley covered the news as soon as the announcement was made, I knew I wanted to learn more about the service and share my findings here.

Rimma Nehme, who demonstrated Cosmos DB in Build's Day 1 keynote, and who is also the product's architect, was kind enough to sit with me last week for an in-depth discussion of the product. Nehme is a computer Science PhD and who previously worked for Microsoft Research. Significantly, she also worked on the SQL Server team for quite some time. If she's gung-ho about Cosmos DB, then I was convinced it was worth knowing more.

What's on deck?
And Nehme is gung-ho, asserting that developers should use Cosmos DB for any new Azure-based application. That's a bold statement, especially for a former member of the SQL Server team who worked on such advanced features as PolyBase (which allows SQL Server to treat files in Hadoop and Azure Blob Storage as if they were local tables) and In-Memory OLTP (originally code-named "Hekaton").

Here are the big reasons Ms. Nehme is such an ardent advocate for Cosmos DB:

• It's battle tested to the max, serving as the database back end-for some of Microsoft's biggest online services
• It's optimized for low-latency database reads and writes. It does this through the use of Solid State Disk storage and with "latchless" and "lockless" data structures which, interestingly, bear some resemblance to those used by SQL Server's In-Memory OLTP model, as well
• It's geo-distributed, across Azure regions/data centers. It's already available in all of them, and will be automatically available in new regions as they come online, because it's a foundational service for Microsoft's own properties
• Developers don't have to worry about the geo-distribution plumbing: data is automatically retrieved at the location closest to the user. What's more, configuring geo-distribution is as simple as clicking on a map to add or drop regions, while the application continues to run. Microsoft aptly calls this feature "global distribution turnkey"
• Unlike other NoSQL databases, which force you into a model of so-called "eventual consistency" with regard to geo-distrubuted propagation of database updates, Cosmos DB allows you to choose between that model, a relational database-like model of strong consistency, or three options in between the two extremes.
• It supports all four NoSQL models (key-value, document, column family and graph)
• Despite the NoSQL categorization, Cosmos DB does support its own dialect of SQL, as well as other APIs (its own JavaScript API and the MongoDB API, with Apache Gremlin and the Azure Table Storage API available in preview, and others on the road map).
• Unlike many NoSQL databases -- or relational databases, for that matter -- which can be stingy with indexing, Cosmos DB indexes every column by default. Developers are free to "opt out" of indexing certain columns, but they needn't "opt-in" to get them.

Taking care of business
Perhaps the biggest reason for Ms. Nehme's enthusiasm for CosmosDB is on the business side (she's got an MBA in addition to her PhD): Microsoft is backing the new database's performance claims with codified service level agreements (SLAs) on its latency, throughput, consistency and high-availability.

This is no mere footnote. As Nehme pointed out to me, Cosmos DB isn't just a technology, it's a service; it was built that way from the start, rather than being a cloud-migrated version of an on-premises product (like Azure SQL Database, which is a cloud-ified version of SQL Server) . Services are businesses, and so business features are core to the platform.

Competitive dynamo
With those SLAs, and Cosmos DB's pricing model and fees (which apparently are the same as those of DocumentDB), Nehme feels that Cosmos DB is very competitive with Amazon Web Services' DynamoDB. The latter, which has no such SLAs, is AWS' foundational NoSQL database service and core to the company's "data gravity" strategy.

The more data customers put in DynamoDB, the more they become beholden to the AWS cloud and the more AWS services they will be likely to use. Microsoft has lacked a strong competitor to DynamoDB: while DocumentDB was somewhat competitive, and sported the adjustable consistency feature present in Cosmos DB, it was a subset of Cosmos DB and lacked much of its versatility. With CosmosDB in place, Microsoft has a horse in the game, not just a pony.

NoSQL, no peace
I have to admit that I find the Cosmos DB platform, if it works as well as advertised, very tempting. But six years ago, I wrote a white paper for Microsoft called NoSQL and the Windows Azure Platform and, while the paper offered a primer on NoSQL technology, it was essentially a refutation of the NoSQL model for use in line-of-business (LOB) applications. The paper described how certain NoSQL features were in any case available on the Microsoft Azure platform and that relational databases, with their consistency guarantees and codified schema, remained the correct choice for most LOB apps.

The paper was essentially commissioned by the SQL Server team, to assert the validity of the relational model that it, along with Oracle, Postgres and MySQL, are based on. While Microsoft obviously paid me a fee, the paper's argument was -- and remains -- consistent with my genuine position on which database models work where.

Can Cosmos "relate?"
While that leaves me in a bit of a quandary with respect to Cosmos DB, one hopeful thought emerges: that one day Cosmos DB, which Microsoft touts as a "multi-mode" database, could support relational operations as well as its current key-value, document, column family and graph modes. In fact, column family databases are effectively schema-based as long as each column family contains only a single value -- something Apache Cassandra takes great advantage of.

If Cassandra can do it, so can Cosmos DB. And if it then adopted a full SQL dialect like SQL Server's Transact-SQL (T-SQL) then, combined with its geo-distribution, low latency and SLAs, it could become very popular among Enterprise developers, as their companies move to the cloud.

And despite Nehme's poker face when I asked her if Microsoft was already thinking about that, I'd be very surprised if it weren't.