Google Apps in first collapse

Just days after launching the paid Premier Edition, Google Apps has fallen over — not for everyone, but for a sizeable minority of Gmail users, who have experienced from 8 to 12 hours of continuous downtime, plus additional knock-on delays to emails that were queued during the downtime. I know I said that how Google handled unplanned downtime incidents was one of the Three megatraps for Google Apps once it had launched paid services, but even I didn't think it would happen as quickly as this. And just as I predicted, Google is handling the glitch with all the familiar disdain and arrogance that global giants routinely show for their small business customers.

There are currently 368 comments and counting in the Gmail Help discussion thread which a Google contributor initiated yesterday under the heading "Gmail errors" but has since been variously renamed by contributors, one of whom picked the sensational: "March 1st 2007 The Day that Will Live On in Infamy" (not only a deliberate echo of the historic reaction to the bombing of Pearl Harbor but also a coincidental nod to my 22nd Feb posting on the day of the Premier Edition launch, Come on down, Google Apps, the price is right, when I said "Today may go down in history as the day when Google started charging for applications.")

The initial message, posted by "Gmail Guide" at 5:50pm GMT (9:50am PST) said:

Hi all,

We are aware of the issues a small sub-set of our users are experiencing when logging in and sending messages. Rest assured that our engineering team is working diligently to resolve these errors. We appreciate your patience.

Please refer back to this thread for updates as they become available.

Although some contributors to the thread are generally supportive of Google, most are critical both of the downtime itself and of the lack of information being provided by Google — users just see a message when attempting to log in that unhelpfully says, "Server Error. We're sorry, but Gmail is temporarily unavailable. We're currently working to fix the problem -- please try logging in to your account in a few minutes." Those users who've managed to find the help forum have had occasional updates from 'Google Guide', who at midnight GMT last night (4pm PST) left the following message:

"This issue should now be resolved. However, some users may continue to experience message delivery delays as the backlog is cleared. We appreciate your patience and thank you for your reports."

Here are some samples of how sore users were still feeling a few hours ago:

"Can you please update? This is awful, i rely on gmail for my business and have been locked out of my account all day?!?!?! WHEN IS THIS GOING TO BE FIXED??!?! AN HOUR OR TWO IS ONE THING BUT 10 ALL DAY IS HORRIBLE. PLEASE HELP!!!!!!"

"I'm still down. My LIFE is on my gmail, and it's been incapacitated for approximately 24 hours. Is there a way for me to be updated on my Gmail's progress? The problem seems specific to certain individuals; how does Gmail know to fix mine in particular?"

This is NOT okay!!!!! I work from home and my e-mail IS my job!!!!!!!!!!!!! ... I have not recieved any e-mail since last night...this is really frusterating. I know they are doing everything they can, and I know things take time and they will do the best that they can, but a timeline would be nice, and some information on WHO to contact if the e-mail is still down after a certain time.

These cries of despair are all too reminiscent of the outcry heard more than a year ago from Salesforce.com users when their application went down for most of Dec 20th, 2005. At the time, I set out a five-point code of practice for on-demand providers. Two of the most important ones were subsequently implemented by Salesforce.com at the excellent trust.salesforce.com. Google has yet to implement this industry best practice:

• Spell out what to do if something does go wrong.
  If the service is suddenly inaccessible, it helps a lot to know there’s another website users can go to for news, or that they’ll receive an email within the hour telling them what’s going on, or that the helpdesk is guaranteed to answer their phone call in less than five minutes — provided, of course, those procedures are robust enough to still work when all hell breaks loose at the provider’s support center!
• Report live service level metrics.
  All customers should be able to view the same dashboard for their services that the provider’s own operations staff get to see. If no such dashboard exists, then isn’t it high time the provider put one in? This is crucial to providing ... accountability.

At least one thing that Google has done is to publish a service level agreement both for Gmail and Google Apps. Experiencing 12 hours of continuous downtime is more than enough to invoke that SLA — although customers will doubtless be astonished to find that 12 hours still amounts to less than two percent of March as a whole and therefore if there are no further problems Google will still be able to claim more than 98% uptime for the month. I wonder though whether Google will honor the 7 days' credit mentioned in its SLA, even though new Google Apps customers are currently in a free trial period.

Will Google honor the spirit of its SLA and extend the free trial period for an extra seven days for affected customers, or will it stick to the letter and refuse on the basis that customers have not started paying for the service yet? I guess that depends on how much Google wants to be taken seriously as an on-demand business service provider and thus how badly it wants to restore its reputation — a reputation that has just taken a very bad knock.

UPDATE [added 2:05 PM]: Google contacted me earlier today to let me know it will credit all its Premier Edition customers with a full month of service, irrespective whether they suffered yesterday's glitch (see Google credits Premier Apps users, restores honor) — a clear signal that Google does indeed mean to be taken very seriously in this market.