Opening the kimono on Salesforce.com's green crystals

Opening the kimono on Salesforce.com's green crystals

Summary: Salesforce.com's chief technology guru Parker Harris, speaking at Cloudforce London this week, mapped out the secret sauce that makes the company's multi-tenancy so special. Why suddenly expose details of something it has always previously kept schtum about?

SHARE:

Last year, a discussion at OnDemand Europe (which this year I'm helping to organize, see disclosure and video) prompted a couple of posts here on the importance and degrees of multi-tenancy. Those postings in turn prompted fellow-Enterprise Irregulars blogger Bob Warfield to liken multi-tenancy to the green crystals apocryphally used in soap marketing:

"... multi-tenancy is much more of a marketing event than a technology event. Whoa! That sort of thing will get me excommunicated from the SaaS Church of Benioff. Well, I’m sorry, but it's true. Multi-tenancy is all about what we used to call 'green crystals marketing' at Borland ... When you're having a hard time differentiating, you find something unique and make it your green crystals. They provide a reason to believe why your offering is better even if they aren't the whole reason or even most of the reason."

Whoa indeed, Bob, I thought at the time, that's a bit extreme. But Bob's words started ringing in my ears the moment Salesforce.com co-founder and chief technology guru Parker Harris stepped up on stage at Cloudforce London this week to pronounce Salesforce.com the leading force in cloud because, quote, "We chose the right algorithm for multi-tenancy." [By the way, I should also mention Salesforce.com is a recent client. See disclosure.]

Harris went on to map out the secret sauce that makes the company's multi-tenancy so special — and I know many readers will want to know what it is, so I'll get to that in a moment, bear with me — but the first question in my mind was, why does Salesforce.com decide that now all of a sudden it wants to start opening up the kimono and exposing details of something it's always previously kept schtum about? ('Green crystals! Green crystals!' whispered Bob's avatar, excitedly jumping up and down on my left shoulder).

"Customers, especially from larger enterprises, have started asking us to tell them more about it," explained EVP of marketing George Hu when I caught up with him as the keynote crowds dispersed. Fact is, I suspect we're going to hear a lot more discussion about multi-tenancy this year. Larger enterprises are looking at cloud technologies and seeing that they can implement private clouds internally, so no wonder many of them are asking what's so special about Salesforce.com. They're probably wondering whether they can replicate what Salesforce.com does on their own private clouds, and sidestep the need to entrust their application infrastructure and data to a third party.

So are Salesforce.com's green crystals just a marketing ploy, or is it really something that customers can't easily replicate in their own private clouds, or by adopting rival cloud platforms? According to Bob Warfield, "Multi-tenancy is ultimately about cost." Merely sharing infrastructure is powerful in itself, as TechCrunch's Erick Schonfield noted in his write-up of a similar presentation at an analyst event in New York last month: All of Salesforce runs on only 1000 servers. But the real key, concluded Bob in his 'green crystals' posting, is to automate the operations surrounding that infrastructure: "Automation is the real driver of SaaS efficiency, but it isn't sexy."

Now I know some people look askance at Salesforce.com's flavor of multi-tenancy. People that I respect in the industry have told me the company only took the route it did because it started out a decade ago, when virtualization was in its infancy and the only option if you wanted all your customers on the same shared application infrastructure was to build multi-tenancy on top of the database. Even Bob, whose SaaS evangelism moves the needles dangerously close to the red areas, says it doesn't make much difference whether you give each customer their own tables within a single shared database as opposed to Salesforce.com's method of running all its customers across just three or four massive shared database tables. In fact, he notes some specific challenges with the Salesforce.com approach, in which every single row of data has to have an extra column that identifies which tenant it belongs to:

".. the cost of the column to identify the tenant is now being charged on every single row of every table. Very likely it isn't a big cost, but it is there. Tables will get bigger too, as all the tenants are piled in. Presumably this can lead to scaling issues sooner."

Well it turns out that this is exactly the challenge that Salesforce.com's 'green crystals' address, so maybe they do have some value that goes beyond mere marketing spin. "We have a patented multi-tenant query optimizer built into Salesforce.com," said Parker Harris in his presentation. "This is one of the key technologies to make it fast." Note the careful inclusion of the word 'patented' to make it clear that this is a variety of green crystal that no other SaaS vendor can use. But what exactly is a 'multi-tenant query optimizer'? I can hear you wondering (if you weren't still interested, you wouldn't have got this far).

I had the good fortune at the post-keynote executive lunch to be sitting next to Trae Chancellor, VP of enterprise strategy, who in his previous role as Salesforce.com's CIO has had unique access to the inner sanctum where the green crystals are kept. He was able to help me figure out how it all works.

It all revolves around special high-speed index tables that are automatically created based on the data that's being stored and how users are interacting with it. Salesforce.com's system is constantly analyzing the queries users are doing and it automatically creates a new index table wherever it sees a query pattern that would work faster if it were indexed. These are not indexes of the whole database, but smaller subsets that group data based on data types or specific categories of data. For every query that runs, the pre-query optimizer then checks out how much data would need to be filtered (by looking at how many rows of data the user has access rights to) and evaluates whether it can use any of the shared indexes to short-cut the operation. It then follows the fastest path to the answer.

This automated query optimization (note that it's performed in software by algorithms, not manually by database administrators) is Salesforce.com's answer to the scaling issues Bob Warfield alluded to in his discussion of multi-tenant databases. Is it a big enough differentiator to evade Bob's 'green crystals marketing' jibe? I'm not enough of a database expert to tell you. What I can say is that computing at this level of sophistication is all about finding the right abstraction. Or as David Wheeler, inventor of the subroutine many decades ago, is quoted as saying, "Any problem in computer science can be solved with another layer of indirection. But that usually will create another problem." Salesforce.com will argue that its method of abstraction at the database layer causes fewer problems than other methods that rely on some form of virtualization — and clearly it believes that its 'multi-tenant query optimizer' makes the worst of those problems go away. Others will say that's just green crystals marketing. In the end — irrespective of who's right and who's wrong according to pure computer science — customers will vote with their spending and the market will decide who wins the argument.

Topic: Enterprise Software

Phil Wainewright

About Phil Wainewright

Since 1998, Phil Wainewright has been a thought leader in cloud computing as a blogger, analyst and consultant.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

7 comments
Log in or register to join the discussion
  • Great Article

    Great article Phil, but it's just another layer of green crystals, LOL.

    The real cost to deliver the service is people, not hardware. Hardware savings through patented query optimizers or any other means is great, but saving incremental ops people through automation is still a much bigger opportunity.

    More on my blog:

    http://smoothspan.wordpress.com/2009/04/10/can-corporate-it-operate-as-efficiently-as-salesforcecom/

    Cheers,

    BW
    BobWarfield
  • RE: Opening the kimono on Salesforce.com's green crystals

    For those of you that are interested, I was hired by Salesforce.com to write a 12 page technical white paper that explains the internals of their multi-tenant architecture. I have a link to the paper in my blog: see http://dbaas.wordpress.com.

    Steve Bobrowski
    spambouncer99
  • RE: Opening the kimono on Salesforce.com's green crystals

    Nicely balanced article, thanks Phil.
    neiltaggart
  • RE: Opening the kimono on Salesforce.com's green crystals

    Multi-tenancey, and it's other-side-of-the-coin sibling "dedicated instances" or "dedicated xxxx" are not capabilities of a service that customers should necessarily need to care about most of the time. They are architectural considerations that certainly impact a SaaS provider's ability to deliver certain service capabilities to a customer. But to must customers they are (or should be) "under the hood" details.

    What SaaS providers (and customers) should be focusing on is their ability to fully meet the customer's business requirements.

    Insofar as a certain architectural detail may commonly be correlated with certain risks (i.e. inabilities to meet a business requirement), i think that's where customers start focusing on architectural details.. I.e. "multi-tenant isn't safe because you can't isolate data", etc.

    kayvaan
  • RE: Opening the kimono on Salesforce.com's green crystals

    A genuine multi-tenant application must have the following characteristics: isolation, security, customizability, recovery, availability, scalability, performance, and provisioning. For most application designers, this is the "multi-tenancy iceberg."

    The discussion here relates to data space sharing and isolation, and how Salesforce.com solved a performance issue caused by their choice for managing multi-tenant data--shared database/shared schema--rather than one of the other two approaches: separate databases and shared database/separate schemas.

    Is this "multi-tenant query optimizer" just "green crystals marketing"? I fully agree with Phil: customers will vote with their spending. But their vote will be driven not only by this particular point, but, above all, by the ability of the provider to completely deal with the multi-tenancy iceberg and make customers believe that they are the software's sole user.

    Bernard Manouvrier
    Chief Architect
    Axway
    BernardManouvrier
  • Multi-Tenant FUD

    <p>Hi Phil,</p>

    <p>Great post. But I think your final point is by far the most salient one, i.e., customer's will decide based on benefits and costs savings. Green crystals are only relevant to the extent that you can tie the technology to a real benefit (like getting things cleaner), and with respect to multi-tenancy the specific technical approach is absolutely irrelevant as long as the vendor <a href="http://chaotic-flow.com/2008/12/08/saas-model-economics-101a-aggregating-customers-for-low-cost-advantage/">realizes strong economies of scale</a> and cost reduction.

    I suspect that Salesforce is pushing it's particular approach to ride the cloud bandwagon and position against Amazon and other cloud "platforms." With Amazon removing any cost advantage one player might have over another from running on a single hardware/OS infrastructure,Salesforce.com is left with emphasizing the idea of running on a single database/application infrastructure. But, the reality is that both software and hardware are increasingly cheap and a SaaS vendor's true advantage comes from standardization, mass customization and scale across the <a href="http://chaotic-flow.com/2009/04/20/saas-tco-the-mirror-image-of-total-cost-of-service/">entire value chain</a>...not just a specific architectural point. Calling out multi-tenancy as a differentiators seems like misguided techno-babble...show me the money!</p>

    <p>By <a href="http://www.crunchbase.com/person/joel-york">Joel York</a></p>
    <p>at <a href="http://chaotic-flow.com">Chaotic Flow</a></p>
    york_joel
  • RE: Opening the kimono on Salesforce.com's green crystals

    Having built SaaS business from ground up and after having sold many SaaS deals myself in last two years, I can testify that customers don't care about technology (i.e. multi-tenancy or anything else for that matter) much. But they do care about quality of service, cost, security, features, performance, ease of use, time to value, SLAs and the list goes on.. They want the best and how its delivered is somewhat irrelevant from end customer?s perspective.

    Having said this, I also know that multi-tenancy does bring a lot to table. Technolgy is an enabler and brings a wow factor.. But SaaS has matured and hence there is no wow factor any more to cash on..It?s back to basic again, IMHO..
    atulkumar