Salesforce.com down...again

Salesforce.com down...again

Summary: I have been getting reports today from salesforce.com customers about ongoing outages.

SHARE:
TOPICS: Salesforce.com
35

I have been getting reports today from salesforce.com customers about ongoing outages. One customer told me that they haven't had a full week since early January without some disruption of service, ranging from not being able to search and delays in sending email to system outages.

salesforceout.jpg

Another customer told me that search was unavailable today from about 8:30 AM to 9:20 AM PST and the Web interface was down from around 10:30 AM to 12:20 PM PST. At the time of this posting, around 2:45 PM PST, a customer said they were getting some server busy responses.

I heard from Bruce Francis, vice president of corporate strategy at salesforce.com, via email around 1:20 PM PST, who said the outage today lasted about 81 minutes.

"At 10:54 am Pacific Time, a primary hardware server in our cluster failed, and one of our North American (NA1) servers did not automatically recover.  This required a manual restart of the NA1 database which completed at 12:15 pm Pacific Time.  All systems are now operational and we apologize for any inconvenience this may have caused."

The new system status page, which provides customers with real time updates on system performance, promised after an earlier outage in December 2005 is being made available to customers this afternoon as well on Monday, Francis also said.

Here is a note sent to customers this morning from co-founder Parker Harris:

To our customers:

On Monday morning Pacific Time January 30, some of our customers experienced intermittent service availability which was caused by a problem in our database cluster.  This issue required Salesforce.com to restart each database instance in the cluster, resulting in a pause in service.  Some customers also experienced a much longer delay in API access during this same time period.  API access was limited in an effort to restore the overall performance of the system.

We very much regret any inconvenience that this may have caused you this week. We believe we have identified the root cause behind the recent availability and performance issues and have updated the service to help avoid a recurrence.

We have been through a lot of change recently, with the new datacenters as part of our Mirrorforce initiative and new software with Winter ’06. While the service is more functional and running faster than at any time in our history, we realize that we need to do more. We have therefore architected both a 30- and 120-day plan for changes in the service to significantly improve availability.  These activities largely include upgrades to software components and installing additional hardware.  This will be our prime focus organizationally as we put these plans into effect. Indeed, we are already executing: this week alone, we have increased our database processing capacity by 50%.  While this additional capacity is unnecessary for normal operation of the service, we believe it will help under certain extreme conditions like losing a database instance under peak load.

We will be keeping you informed of the updates to our systems through a series of communications over the next few weeks and months.  We look forward to sharing more details of our plans as they are implemented.

Nothing is more important to us than your trust as a customer and we are working harder than ever to earn your trust every day.

Yours,

Parker Harris
Co-founder and EVP, Technology 

The disclosure from Bruce Francis--which doesn't exactly agree with reports I received from customers--is a good change of pace from the previous outage on January 30, when the company wasn't forthcoming with details other than to say it was minor and lasted 30 minutes, which wasn't the case for customers who had intermittent access and couldn't access the salesforce.com API for hours. A bit of Clintonese, I suspect, in how "minor" (sex in Clinton's case) was defined by CEO Marc Benioff.

benioff2001.jpgThe "force" is still with Benioff and his friend Yoda, but customers will only tolerate so much inconvenience and business disruption. One of the good traits of software as a service (SaaS) is the switching costs are generally low--take your data and go somewhere else. There are plenty of SaaS CRM options. Salesforce.com isn't alone in suffering scaling pains as it grows--it goes with SaaS territory. Those that have more consistent performance and reliability will be the winners... 

Topic: Salesforce.com

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

35 comments
Log in or register to join the discussion
  • Nah, can't be ;-)

    Hey, it's probably your imagination ;-). At least that's what Salesforce.com said last month.
    george_ou
  • Exactly why SAAS is doomed to failure.

    Software as a service is the latest mantra of those who want you to pay forever (ie: rent) your software. They have mounted a constant media barrage about their "better" way of doing business, but here is the proof that the skeptics fears are real.

    Your data sits on someone else's server, and is accessible only at *their* whim. If Salesforce is inaccessible, whether through server or network failure or through criminal intent (such as a DDoS attack), your out of business.

    Worse still, if Salesforce's server is hacked, your proprietary business data is in the hands of someone else and could be sold to your competitor. Industrial espionage is already bad enough, why make it worse by losing control of the storage medium upon which your critical business data resides?

    It was only a matter of time before the truth popped up to dispel the SAAS lies.

    Regards,
    Jon
    JonathonDoe
    • In my mind, it was doomed before it started...

      SaaS, in my opinion, is nothing more than a money grab. A guaranteed revenue stream for those that provide the so-called "service".

      They can't substantially innovate any longer, so this just a switch in business models due to concerns for revenue. It really offers very little to the end user, at least in most cases.

      To depend on someone else's equipment, online status and ability to secure the critical data used in mission critical applications is ludicrous. Of course, there will those that grab for the carrot on the stick because of the supposed lower cost, but I hope they don't have the audacity to whine when their business is damaged or their productivity drops because of their vendor's foul-up.
      shawkins
    • RE: Salesforce.com down...again

      @JonathonDoe [url=http://www.youtubetomp4converter.info]YouTube To MP4 Converter [/url]
      [url=http://www.pdfconverterformac.org]Pdf Converter for Mac[/url]
      [url=http://www.youtubeconvertermac.org]YouTube Converter Mac[/url]
      soogyzauwk
  • Where can I sign up for this superb SAAS Stuff?...

    I need a vacation. Looks like the best way to get one.
    BitTwiddler
    • I can server your needs

      Apparently all I need to do is sell you then go offline a lot...I can handle that.
      IT Scion
  • Stick a fork in 'em

    They're done. Besides there are systems you can run for less yourself. Don't have to depend on another company holding all your client data hostage.
    Chad_z
    • salesforce.com's modus operandi

      I don't know much about managing an internet business, but salesforce is not the business model I would adopt. Simply, I don't know what their motives for the site. Your thoughts..
      mlemmons@...
  • Hey, who needs Windows apps anyway?

    It's al about Web 2.0! This is just a one time problem! Web 2.0 will conquer all. I'm going to throw out all of my full-blown Windows applications and replace them with AJAX applications! Oh, wait. This is the second time in two weeks? Oh, wait. Their performance even when they're online is crap. Maybe I'll stick to good old Windows applications in stead of putting all of my eggs/apps into the Web-only basket.
    marksashton
    • Misplaced ire...

      It's not whether its a web app vs. a Windows app that's the issue. We do lots of web apps in my company, and host them all internally - our own app servers and databases. We get the benefits of low-deployment and maintenance costs, coupled with the benefits of retaining control. I still see some value in the richness of the GUI with Windows apps. You still can't get the same nice feel with a web app. OTOH, web apps have administrative benefits. SAAS has problems - you lose control - but don't get caught trying to assert that the issue is "web apps" in general - that dog won't hunt.
      Techboy_z
      • Well said

        I'm not suggesting that it is an either/or proposition. There are many advantages of Web-based applications as you outline. But the CEO of Salesforce.com has said publicly that he see's a world of Web only applications. I think that's rediculous for many rasons:

        1. Why waste all of the processing power/storage on the client? There is more pure processing power on client PC's than in all of the world's servers combined. Not using that power is a waste.

        2. All of that power can deliver a better user experience in many cases than Web-based apps. The challenge on the client is to make the management and deployment of client-side applications better. There is progress. MSFT's "click once" deployment approach, for example, when properly implemented offers the best of both worlds: a rich client application with the ease of deployment and updating of Web-based apps.

        3. Although connectivity is becoming more pervasive it will be a VERY long time if ever that it is 100% reliable. There is still a need for users to be able to use their appliations offline.

        I could go on but you get the idea. The people who are/have jumped on the Web 2.0/Ajax bandwagon to the point that they ignore these points do so out of naivity.
        marksashton
        • Web apps or Windows apps. It depends on the project requirements.

          Web apps are not the silver bullet nor are Windows apps. It depends on the project requirements. The same goes for languages. C, C++, C#. It depends on the project. I think for a great many projects, smart clients will be the way to go. Smart clients give you some of the goodness of web apps and some of the goodness of Windows apps. Once again, project requirements. It ain't rocket science.

          -Splekem
          Jake Danger
  • why does ZDNet care so much?

    This is a honest question.
    There are plenty of 'Web 2' sites that go down, why does this one get so much coverage, specifically by ZDNet?
    corticus
    • Recognizing the "next big thing"

      ZDNet has annointed this case as a test to whether software as a service can become huge or not. With all eyes on SalesForce, many different sectors are watching to see if they should move in this direction. The ability to kill Oracle & SAP's business model is considered big news.

      Do a quick search on SalesForce on ZDNet and you'll get a flavor for what importance they put on this effort.
      ejhonda
  • Salesforce.com down...again

    Unfortunately, my company recently signed on as a new Salesforce customer, with lots of mission critical data hosted there. Outages like these seriously make me question the decision to go with them. The situation is sort of ironic actually, because if anyone has been in a Salesforce pitch meeting before, one of their major selling points is all the redundancy measures in place to prevent outages.
    jdd48
  • weeeeeeeeee!!!

    Fun fun..
    GrindinAxTaRupert
  • classic, unfortunate, down again

    I have been around some, since the earliest days of computing.
    It really amazes me sometimes when I hear of things like this.
    The lot of you reading these articles really believe things like "no
    system is perfect, and all have downtime."

    Not when they are put together correctly. Not when you spend
    enough on your hardware. I am not spelling out brand names
    here, but I have personally seen general purpose computing
    machines running after 10 years. Not just no downtime, but
    nothing lost to maintenance!

    The only reason salesforce.com has these problems is that they
    will not spend enough to solve it.
    gjsherr
  • Remember Mainframes

    Centralized services are doomed to failure. The reason we all switched to PCs was the unreliable nature of mainframes and the massive impact their frequent crashes had on all of us. The "try again later" message is just like the "try again later" messages I would see on my old 3278 terminal waiting for the IT staff to reboot the IBM mainframe.
    WindowsDev
    • Right on the mark

      I always wonder why these guys try to take a failed model and revive it with a model that's even worse.
      tshinder@...
    • Other reasons mainframes "died"

      We didn't switch from mainframes because they were unreliable - the ones I worked on were rock-solid. We switched because CA, IBM, etc... were killing us on license costs, and because no one coming out of school wanted to learn COBOL. The evolution was fairly "organic", and now we're seeing the next wave. TCO will still be cheaper - how much does a large organization spend on desktop support (including software installs, fixes, etc...)? "On demand" is the wave of the future - get on board :o)
      dmIlovesw