Salesforce.com customers continue to suffer as instances of the social enterprise service crumbled in the face of a power outage, the company said today.
The bright side is that many were tucked up in bed following the initial outage that began at 12:49 a.m. PDT. Europeans picked up the slack as they headed into work around the same time on local time. Many took to Twitter to complain.
Even the status page was down for a while, some said on Twitter. Power appears to be the cause of the outage, as explained on the site's status page --- once it jumped back into life:
2:46 am PDT : NA1/NA5/NA6/CS0,CS3,CS1,CS12 salesforce.com System Status
The salesforce.com NA1/NA5/NA6/CS0,CS3,CS1,CS12 instances are continuing to experience a service disruption. Power issues were detected but our technician onsite has confirmed this has been fixed. We are currently working to restore the service. Please check the status of trust.salesforce.com frequently for updates regarding this issue.
Since then, the company noted that "standard salesforce.com reporting, contacts, updates, case entry services" were available, but some will see problems with "sporadic search and file attachment performance."
The Salesforce.com application store is also down, the company said at 5:30 a.m. PDT.
It's the second such major outage in as many weeks.
Late June saw the last major outage, in which a fault occurred in its storage tier. Performance suffered in both North America and European regions. In total, the outage lasted seven hours during European business primetime.
At the time of writing, service had been restored on NA1 and NA5 servers --- though performance was still choppy --- but gave no estimates were given on the restoration of the remaining five affected instances: NA6, CS0, CS3, CS1 and CS12.
Hats off to the Salesforce public relations team. The status site remains (mostly) up and provided detailed information, and there were at times a tweet a minute from its Twitter-based team to keep the spirits up among those who were suffering in the outage.
ZDNet contacted Salesforce for comment, but did not receive a reply at the time of writing.
Update at 11:20 a.m. PDT: All of the instances are back up with the exception of CS12. While warnings and notifications are still appearing under all of all instances previously affected --- such as issues relating to search functionality --- many can at least now log back into the Salesforce.com website.
"The salesforce.com Technology Team is working to fail the CS12 instance over to its DR datacenter," the update at 11 a.m. PDT read.
At the time of this update, the service should have all instances up by 12 midday PDT.
Update at 12:15 p.m. PDT: The status page says Salesforce has paused the CS12 failover to its alternate datacenter but says it could restore the instance "within the same time window which is a lower risk option."
It also added:
11:30 am PDT : NA5/CS1 Search and CS12 Outage System Status - Update
The salesforce.com Technology Team has identified and repaired a problem impacting NA5 search for large indexes. The search servers, when restarted, defaulted to a sub-optimal packet size setting on the host network interface card. As such, large indexes were broken up and handled inefficiently. We found and corrected the packet size setting and remediated the search performance impact for NA5.
Update at 12:30 p.m. PDT: Salesforce said it has "resolved all issues with the CS12 instance, and all CS12 services are restored." All instances are back up and running, though some problems with search will persist throughout the remainder of the day.