Google: Gmail outage caused by software update

Summary:The web giant has turned to tape backups to restore email access for thousands of users left stranded after a storage software update caused an outage on Monday

A storage software update was behind a Gmail outage on Monday, according to Google.

The update left users of the service unable to log in or met by a blank inbox, with all messages, attachments and chat logs apparently having been deleted. Despite having several datacentres that handle the back-end for the mail service, the issue was heightened by backup copies also being compromised, the company said in a blog post on Tuesday.

Gmail outage update

The Gmail outage was caused by a storage software update, with Google working to restore affected users. Credit: Google

"When we discovered the problem, we immediately stopped the deployment of the new software and reverted to the old version," Ben Treynor, who heads up engineering and site reliability for Google, said on the blog.

Recovering lost emails

The problems have largely been resolved, Google said, adding that it is trying to recover the lost emails. However, it noted that the restoration process has been hampered by the "unusual" bug which had compromised backups stored in multiple datacentres. As a result, it had been restoring emails from offline tape backups.

"Since the tapes are offline, they're protected from such software bugs," Treynor said. "But restoring data from them also takes longer than transferring your requests to another datacentre, which is why it's taken us hours to get the email back instead of milliseconds."

On Monday, Google estimated that 0.08 percent of its user base had been affected, but on Tuesday, it revised the estimate to 0.02 percent, roughly equating to 38,000 accounts.

In May 2009, the Google mail service suffered an outage that prevented approximately 14 percent of its user base from logging on or using the service normally.

The company updated its App status dashboard at 9.15am GMT on Tuesday to say that service has been restored for some users, and that service for all users should be back online within 10 hours. However, it also said that the time frame is an estimate and could change.


Get the latest technology news and analysis, blogs and reviews delivered directly to your inbox with ZDNet UK's newsletters.

Topics: Hardware

About

With a psychology degree under his belt, Ben set off on a four-year sojourn as a professional online poker player, but as the draw of the gambling life began to wane his attentions turned to more wholesome employment.With several years' experience covering everything in the world of telecoms and mobility, Ben's your man if it involves a s... Full Bio

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Related Stories

The best of ZDNet, delivered

You have been successfully signed up. To sign up for more newsletters or to manage your account, visit the Newsletter Subscription Center.
Subscription failed.