Microsoft shares details of holiday Hotmail outage

Microsoft Hotmail hiccuped around the New Year holiday, resulting in approximately 17,000 users finding some or all of their messages missing.

At the time, Microsoft officials declined to comment on what happened. But on January 6, they shared more details about the causes of the recent Hotmail outage that started on December 30. From a post on the "Inside Windows Live" blog:

"On December 30th, we had an error in a script that inadvertently removed the directory records of a small number of real user accounts along with a set of test accounts. Please note that the email messages and folders of impacted users were not deleted; only their inbox location in the directory servers was removed. Therefore when they logged in, a new mailbox was automatically created for them on a new storage server that didn’t contain their old messages and folders. This is why the accounts received the “Welcome to Hotmail” message."

The post notes that even thought he first ticket about the outage was received December 30, there were problems isolating the cause. On January 1, Microsoft escalated the priortiy of the ticket, following continued reports.

"Our first step was to restore these users’ entries in the directory servers, which we did by early on the morning of January 2 PST. We then merged their old email messages and folders with any new mail they’d received throughout the day on January 2nd. This required multiple passes to capture all the accounts and messages, so for some users, service wasn’t completely restored until January 5th. We completed the merge for 16,035 users on January 2nd and by January 5th had completed this for the remaining 1,320 users who were affected by this particular issue."

(Microsoft officials said on January 3 that the issues had been resolved, but requested that users who still had problems alert the Hotmail team.)

Microsoft officials said there was no permanent loss of any user data for the affected accounts. However, "the only unfortunate exception to this statement is that, if you were affected by this incident and you didn’t sign in to your account between the time of the incident and the time your account was restored, then any messages sent to your account during that time would have bounced," officials noted.

To prevent similar Hotmail problems going forward, the team is updating its infrastructure "to use a separate code path for provisioning and removing test accounts, so that our testing no longer risks affecting real user accounts," said officials. In addition, Microsoft is changing its issue alert process to assign higher priorities to issues reported by multiple users.

Microsoft officials noted they received unrelated data-loss issue reports from people who set up a POP client on their computer or mobile phone. (I may have been one of those reports.) Microsoft suggests users who still think they're missing email from their Hotmail accounts to first check this Solution Center article about common reasons for missing Hotmail. If that doesn't address the problem, Microsoft is asking users to report their issues via the Hotmail Solution Center Forums.