A storage software update was behind a Gmail outage on Monday, according to Google.
The update left users of the service unable to log in or met by a blank inbox, with all messages, attachments and chat logs apparently having been deleted. Despite having several datacentres that handle the back-end for the mail service, the issue was heightened by backup copies also being compromised, the company said in a blog post on Tuesday.
The Gmail outage was caused by a storage software update, with Google working to restore affected users. Credit: Google
"When we discovered the problem, we immediately stopped the deployment of the new software and reverted to the old version," Ben Treynor, who heads up engineering and site reliability for Google, said on the blog.
Recovering lost emails
The problems have largely been resolved, Google said, adding that it is trying to recover the lost emails. However, it noted that the restoration process has been hampered by the "unusual" bug which had compromised backups stored in multiple datacentres. As a result, it had been restoring emails from offline tape backups.
"Since the tapes are offline, they're protected from such software bugs," Treynor said. "But restoring data from them also takes longer than transferring your requests to another datacentre, which is why it's taken us hours to get the email back instead of milliseconds."
On Monday, Google estimated that 0.08 percent of its user base had been affected, but on Tuesday, it revised the estimate to 0.02 percent, roughly equating to 38,000 accounts.
In May 2009, the Google mail service suffered an outage that prevented approximately 14 percent of its user base from logging on or using the service normally.
The company updated its App status dashboard at 9.15am GMT on Tuesday to say that service has been restored for some users, and that service for all users should be back online within 10 hours. However, it also said that the time frame is an estimate and could change.
Get the latest technology news and analysis, blogs and reviews delivered directly to your inbox with ZDNet UK's newsletters.