Opening the kimono on Salesforce.com's green crystals

Last year, a discussion at OnDemand Europe (which this year I'm helping to organize, see disclosure and video) prompted a couple of posts here on the importance and degrees of multi-tenancy. Those postings in turn prompted fellow-Enterprise Irregulars blogger Bob Warfield to liken multi-tenancy to the green crystals apocryphally used in soap marketing:

"... multi-tenancy is much more of a marketing event than a technology event. Whoa! That sort of thing will get me excommunicated from the SaaS Church of Benioff. Well, I’m sorry, but it's true. Multi-tenancy is all about what we used to call 'green crystals marketing' at Borland ... When you're having a hard time differentiating, you find something unique and make it your green crystals. They provide a reason to believe why your offering is better even if they aren't the whole reason or even most of the reason."

Whoa indeed, Bob, I thought at the time, that's a bit extreme. But Bob's words started ringing in my ears the moment Salesforce.com co-founder and chief technology guru Parker Harris stepped up on stage at Cloudforce London this week to pronounce Salesforce.com the leading force in cloud because, quote, "We chose the right algorithm for multi-tenancy." [By the way, I should also mention Salesforce.com is a recent client. See disclosure.]

Harris went on to map out the secret sauce that makes the company's multi-tenancy so special — and I know many readers will want to know what it is, so I'll get to that in a moment, bear with me — but the first question in my mind was, why does Salesforce.com decide that now all of a sudden it wants to start opening up the kimono and exposing details of something it's always previously kept schtum about? ('Green crystals! Green crystals!' whispered Bob's avatar, excitedly jumping up and down on my left shoulder).

"Customers, especially from larger enterprises, have started asking us to tell them more about it," explained EVP of marketing George Hu when I caught up with him as the keynote crowds dispersed. Fact is, I suspect we're going to hear a lot more discussion about multi-tenancy this year. Larger enterprises are looking at cloud technologies and seeing that they can implement private clouds internally, so no wonder many of them are asking what's so special about Salesforce.com. They're probably wondering whether they can replicate what Salesforce.com does on their own private clouds, and sidestep the need to entrust their application infrastructure and data to a third party.

So are Salesforce.com's green crystals just a marketing ploy, or is it really something that customers can't easily replicate in their own private clouds, or by adopting rival cloud platforms? According to Bob Warfield, "Multi-tenancy is ultimately about cost." Merely sharing infrastructure is powerful in itself, as TechCrunch's Erick Schonfield noted in his write-up of a similar presentation at an analyst event in New York last month: All of Salesforce runs on only 1000 servers. But the real key, concluded Bob in his 'green crystals' posting, is to automate the operations surrounding that infrastructure: "Automation is the real driver of SaaS efficiency, but it isn't sexy."

Now I know some people look askance at Salesforce.com's flavor of multi-tenancy. People that I respect in the industry have told me the company only took the route it did because it started out a decade ago, when virtualization was in its infancy and the only option if you wanted all your customers on the same shared application infrastructure was to build multi-tenancy on top of the database. Even Bob, whose SaaS evangelism moves the needles dangerously close to the red areas, says it doesn't make much difference whether you give each customer their own tables within a single shared database as opposed to Salesforce.com's method of running all its customers across just three or four massive shared database tables. In fact, he notes some specific challenges with the Salesforce.com approach, in which every single row of data has to have an extra column that identifies which tenant it belongs to:

".. the cost of the column to identify the tenant is now being charged on every single row of every table. Very likely it isn't a big cost, but it is there. Tables will get bigger too, as all the tenants are piled in. Presumably this can lead to scaling issues sooner."

Well it turns out that this is exactly the challenge that Salesforce.com's 'green crystals' address, so maybe they do have some value that goes beyond mere marketing spin. "We have a patented multi-tenant query optimizer built into Salesforce.com," said Parker Harris in his presentation. "This is one of the key technologies to make it fast." Note the careful inclusion of the word 'patented' to make it clear that this is a variety of green crystal that no other SaaS vendor can use. But what exactly is a 'multi-tenant query optimizer'? I can hear you wondering (if you weren't still interested, you wouldn't have got this far).

I had the good fortune at the post-keynote executive lunch to be sitting next to Trae Chancellor, VP of enterprise strategy, who in his previous role as Salesforce.com's CIO has had unique access to the inner sanctum where the green crystals are kept. He was able to help me figure out how it all works.

It all revolves around special high-speed index tables that are automatically created based on the data that's being stored and how users are interacting with it. Salesforce.com's system is constantly analyzing the queries users are doing and it automatically creates a new index table wherever it sees a query pattern that would work faster if it were indexed. These are not indexes of the whole database, but smaller subsets that group data based on data types or specific categories of data. For every query that runs, the pre-query optimizer then checks out how much data would need to be filtered (by looking at how many rows of data the user has access rights to) and evaluates whether it can use any of the shared indexes to short-cut the operation. It then follows the fastest path to the answer.

This automated query optimization (note that it's performed in software by algorithms, not manually by database administrators) is Salesforce.com's answer to the scaling issues Bob Warfield alluded to in his discussion of multi-tenant databases. Is it a big enough differentiator to evade Bob's 'green crystals marketing' jibe? I'm not enough of a database expert to tell you. What I can say is that computing at this level of sophistication is all about finding the right abstraction. Or as David Wheeler, inventor of the subroutine many decades ago, is quoted as saying, "Any problem in computer science can be solved with another layer of indirection. But that usually will create another problem." Salesforce.com will argue that its method of abstraction at the database layer causes fewer problems than other methods that rely on some form of virtualization — and clearly it believes that its 'multi-tenant query optimizer' makes the worst of those problems go away. Others will say that's just green crystals marketing. In the end — irrespective of who's right and who's wrong according to pure computer science — customers will vote with their spending and the market will decide who wins the argument.