Why multi-tenancy matters

I'd like to refine the description of multi-tenancy that I set out in my post earlier this week on Many degrees of multi-tenancy. Intacct's CTO, Aaron Harris, responded with some very illuminating detail on the vendor's architecture, which leads me to reclassify Intacct (and indeed NetSuite) into the top tier alongside Salesforce.com. Their differences come down to implementation choices rather than any difference of principle, as I'll explain. There's also been lots of thoughtful TalkBack, and I'll highlight some of those comments too.

At the top tier of multi-tenancy, the debate is merely about how many identical instances a vendor wants to support. There's universal agreement on the need for every customer to be running on identical code, right down to shared schema at the database level. The debate revolves around the optimum size and number of instances that the vendor replicates within its data center. Intacct's Harris (pictured here) succinctly set out the main factors in an email to me earlier this week, which I'm quoting with permission:

"Intacct currently runs all 2,500 of its customers on 10 production databases. Roughly three years ago we exceeded the capacity of a single DB instance and were forced to make a decision about how we were going to scale. Our options were:

1. go with a big-iron, rack approach similar to Salesforce at the time (scale up),
2. go with many inexpensive servers similar to NetSuite (scale out) or
3. continue to buy the most powerful Intel-based boxes we could find, and add more as necessary.

"We ruled out a) for a couple reasons. We felt having a single instance causes too much exposure. Even if that single instance has greater reliability numbers, the outages become too visible. Second, our over-arching design principle for operations is 'simple, rugged, economical design'. Rack fails on 'simple' and 'economical'.

"We ruled out b) because we feel having many DB servers eliminates one of the core reasons for multi-tenancy — economics. It's not the acquisition of lots of inexpensive servers that fails the economics test — it's the increased burden on operations staff. Additionally, more DBs means more risk that your instances won't be identical."

That final point brings us onto the topic of upgrades, which requires some elaboration. First of all here's what Harris wrote in his email:

"Intacct is religious about upgrading all customers at the same time. We will never tolerate running multiple versions of our code. Again, this gets back to why we go multi-tenant in the first place — economics. Each version adds more regression testing, more support complexities, and adds unnecessary services processes. Similarly, even though customer data is leveled across multiple databases, the databases are identical at the hardware, system, and schema level. We can easily move customers from one DB to another (which does happen as we level load) because our DBs are identical."

At first glance, that seems in complete contradiction to what former CEO David Thomas said last week, that Intacct allows its customers to choose when to upgrade to new releases. The missing factor that explains the apparent conflict is that although the code is upgraded for everyone at the same time, the new features are implemented as configuration options that customers can choose to 'switch on' as needed. Of course there are exceptions where the new code makes changes that can't be optional, but these are in the minority, as Harris explained to me in a phone call yesterday:

"The bulk of new functionality is only available if you enable it or configure it," said Harris. "It's very rare that we release something that we would call disruptive — for example, changing the position of a field from the bottom to the top of a screen."

Harris said that Intacct makes feature releases every month, each with six to twelve new features, but that in the past year, only six of those dozens of features had been 'disruptive' changes where customers had no choice about implementing them. Intacct has a process that gives users plenty of notice for this type of change, which typically involves putting an alert on the screen next to the feature, along with a clickable link that takes the user to further detail about the planned change and why it's being made.

Harris also noted the difference between "marketing releases," where vendors hold a big event and launch a new release with a lot of new features, and more routine upgrade releases that are often not publicized outside of the user base: "We [SaaS vendors] all have the capability of doing releases on a more frequent basis. There's a difference between doing a launch that we publicize and doing a release where you iteratively enhance and improve the product."

That ability to do frequent iterative releases again comes back to the low management overhead of running replicated shared-schema instances, in contrast to lesser degress of multi-tenancy that permit variations between separate instances. Several TalkBack commenters picked up on the economic and operational impact of low management overhead as one of the key characteristics of a true multi-tenant SaaS model. John Martin, for example, warned readers to beware of 'fake-SaaS' offerings:

"Customers should ask carefully about how many versions a provider maintains, how many instances/infrastructures, and how customizations are done, to ensure they get full value from the SaaS solution and aren't eventually orphaned by fake-SaaS providers."

Another TalkBack commenter, mmahal, complained that running separate instances in 'pods', as recommended by Oracle, was simply a throwback to the SoSaaS days of application service providers (ASPs): "Oracle hosting is just ASP on juice!" he wrote. [Disclosure: I was a paid keynote speaker at an Oracle SaaS partner event in January. Salesforce.com, among other SaaS vendors, is also a recent client].

Others, such as saas-guy, were more understanding of Oracle's position that software vendors should be free to choose between different models:

"When designing the right architectural approach for your SaaS offering, Which degree of multi-tenancy is right for you depends on the cost of re-writing or updating your software, cost of technology, cost of maintenance & support of the infrastructure, along with your customer needs," he wrote. "Each would-be SaaS company has to look at their needs and decide on the right approach for them. Oracle is agnostic and wants to educate their partners and make sure they understand the pros & cons with each approach."

I think it's true that vendors have to weigh up the costs and time-to-market impact of rearchitecting their applications for multi-tenancy. There are arguments for using virtualization as a short-cut that gets a SaaS offering to market faster (as part of the journey to SaaS). But vendors need to be equally aware of what they're giving up by going down that route. I'm not convinced that Oracle's 'agnosticism' on that score is as even-handed as its friends would have us believe — even though Intacct, NetSuite and Salesforce.com all run their SaaS infrastructures on Oracle databases. I think a lot of people coming from outside the SaaS world believe that multi-tenancy is overrated. Intacct's Aaron Harris has helped illuminate why so many of the most experienced SaaS providers believe multi-tenancy is fundamental to what they do.