Salesforce.com down...again

I have been getting reports today from salesforce.com customers about ongoing outages. One customer told me that they haven't had a full week since early January without some disruption of service, ranging from not being able to search and delays in sending email to system outages.

Another customer told me that search was unavailable today from about 8:30 AM to 9:20 AM PST and the Web interface was down from around 10:30 AM to 12:20 PM PST. At the time of this posting, around 2:45 PM PST, a customer said they were getting some server busy responses.

I heard from Bruce Francis, vice president of corporate strategy at salesforce.com, via email around 1:20 PM PST, who said the outage today lasted about 81 minutes.

"At 10:54 am Pacific Time, a primary hardware server in our cluster failed, and one of our North American (NA1) servers did not automatically recover.  This required a manual restart of the NA1 database which completed at 12:15 pm Pacific Time.  All systems are now operational and we apologize for any inconvenience this may have caused."

The new system status page, which provides customers with real time updates on system performance, promised after an earlier outage in December 2005 is being made available to customers this afternoon as well on Monday, Francis also said.

Here is a note sent to customers this morning from co-founder Parker Harris:

To our customers:

On Monday morning Pacific Time January 30, some of our customers experienced intermittent service availability which was caused by a problem in our database cluster.  This issue required Salesforce.com to restart each database instance in the cluster, resulting in a pause in service.  Some customers also experienced a much longer delay in API access during this same time period.  API access was limited in an effort to restore the overall performance of the system.

We very much regret any inconvenience that this may have caused you this week. We believe we have identified the root cause behind the recent availability and performance issues and have updated the service to help avoid a recurrence.

We have been through a lot of change recently, with the new datacenters as part of our Mirrorforce initiative and new software with Winter ’06. While the service is more functional and running faster than at any time in our history, we realize that we need to do more. We have therefore architected both a 30- and 120-day plan for changes in the service to significantly improve availability.  These activities largely include upgrades to software components and installing additional hardware.  This will be our prime focus organizationally as we put these plans into effect. Indeed, we are already executing: this week alone, we have increased our database processing capacity by 50%.  While this additional capacity is unnecessary for normal operation of the service, we believe it will help under certain extreme conditions like losing a database instance under peak load.

We will be keeping you informed of the updates to our systems through a series of communications over the next few weeks and months.  We look forward to sharing more details of our plans as they are implemented.

Nothing is more important to us than your trust as a customer and we are working harder than ever to earn your trust every day.

Yours,

Parker Harris
Co-founder and EVP, Technology 

The disclosure from Bruce Francis--which doesn't exactly agree with reports I received from customers--is a good change of pace from the previous outage on January 30, when the company wasn't forthcoming with details other than to say it was minor and lasted 30 minutes, which wasn't the case for customers who had intermittent access and couldn't access the salesforce.com API for hours. A bit of Clintonese, I suspect, in how "minor" (sex in Clinton's case) was defined by CEO Marc Benioff.

The "force" is still with Benioff and his friend Yoda, but customers will only tolerate so much inconvenience and business disruption. One of the good traits of software as a service (SaaS) is the switching costs are generally low--take your data and go somewhere else. There are plenty of SaaS CRM options. Salesforce.com isn't alone in suffering scaling pains as it grows--it goes with SaaS territory. Those that have more consistent performance and reliability will be the winners...