Microsoft shares details of holiday Hotmail outage

Microsoft shares details of holiday Hotmail outage

Summary: Microsoft Hotmail hiccuped around the New Year holiday, resulting in approximately 17,000 users finding some or all of their messages missing. At the time, Microsoft officials declined to comment on what happened. But on January 6, they shared more details about the causes of the recent Hotmail outage.

SHARE:

Microsoft Hotmail hiccuped around the New Year holiday, resulting in approximately 17,000 users finding some or all of their messages missing.

At the time, Microsoft officials declined to comment on what happened. But on January 6, they shared more details about the causes of the recent Hotmail outage that started on December 30. From a post on the "Inside Windows Live" blog:

"On December 30th, we had an error in a script that inadvertently removed the directory records of a small number of real user accounts along with a set of test accounts. Please note that the email messages and folders of impacted users were not deleted; only their inbox location in the directory servers was removed. Therefore when they logged in, a new mailbox was automatically created for them on a new storage server that didn’t contain their old messages and folders. This is why the accounts received the “Welcome to Hotmail” message."

The post notes that even thought he first ticket about the outage was received December 30, there were problems isolating the cause. On January 1, Microsoft escalated the priortiy of the ticket, following continued reports.

"Our first step was to restore these users’ entries in the directory servers, which we did by early on the morning of January 2 PST. We then merged their old email messages and folders with any new mail they’d received throughout the day on January 2nd. This required multiple passes to capture all the accounts and messages, so for some users, service wasn’t completely restored until January 5th. We completed the merge for 16,035 users on January 2nd and by January 5th had completed this for the remaining 1,320 users who were affected by this particular issue."

(Microsoft officials said on January 3 that the issues had been resolved, but requested that users who still had problems alert the Hotmail team.)

Microsoft officials said there was no permanent loss of any user data for the affected accounts. However, "the only unfortunate exception to this statement is that, if you were affected by this incident and you didn’t sign in to your account between the time of the incident and the time your account was restored, then any messages sent to your account during that time would have bounced," officials noted.

To prevent similar Hotmail problems going forward, the team is updating its infrastructure "to use a separate code path for provisioning and removing test accounts, so that our testing no longer risks affecting real user accounts," said officials. In addition, Microsoft is changing its issue alert process to assign higher priorities to issues reported by multiple users.

Microsoft officials noted they received unrelated data-loss issue reports from people who set up a POP client on their computer or mobile phone. (I may have been one of those reports.) Microsoft suggests users who still think they're missing email from their Hotmail accounts to first check this Solution Center article about common reasons for missing Hotmail. If that doesn't address the problem, Microsoft is asking users to report their issues via the Hotmail Solution Center Forums.

Topics: Browser, Collaboration, Microsoft, Outage

About

Mary Jo has covered the tech industry for 30 years for a variety of publications and Web sites, and is a frequent guest on radio, TV and podcasts, speaking about all things Microsoft-related. She is the author of Microsoft 2.0: How Microsoft plans to stay relevant in the post-Gates era (John Wiley & Sons, 2008).

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

30 comments
Log in or register to join the discussion
  • RE: Microsoft shares details of holiday Hotmail outage

    dsxcvxc
    sdjhfksdjhfkj
  • Thank you for your honest reporting. Too bad Apple isn't so honest

    Apple has offered no further explanations for why all the alarm clocks in all their iOS devices failed starting Jan 1, 2011 and while we have now read in great deal on ZDNet all about this hotmail issue, interesting that the iOS alarm bug, something that affected millions of users, received one tiny note from AKH on Jan 1, no notes from the Apple Core, and absolutely no follow up from anyone.

    No wonder people think Apple products "Just Work" considering no one here reports on Apple products when they "Don't Just Work".
    NonZealot
    • RE: Microsoft shares details of holiday Hotmail outage

      @NonZealot ... I heard of 7 coworkers call in "late" for their shifts at our hospital over the holiday weekend, including Monday the 3rd. Yes, since then they've been magically working again, but telling time on a computer has been something that we've been doing for years now. Shouldn't an iWhatever be able to do it?

      I've been laughing about this bug all week! Where did I hear about it? From some random blog I've never heard of via a friend on Facebook. Go figure.
      GoodThings2Life
    • Not exactly a huge Hotmail outage either

      @NonZealot
      17,000 users out of 360M = 0.0047% of users. Must have been a slow news day.

      How many iPhones have the alarm clock bug? All of them.
      dazzlingd
      • And yet...

        100% of the hotmail users affected use Hotmail for email. I don't know what percentage of iPhone users rely on their phone to be their alarm clock, but I'd imagine it's in the same ballpark of 0.0047%.
        jasonp@...
    • RE: Microsoft shares details of holiday Hotmail outage

      @NonZealot Yet you have an iPhone.

      Cue the double standard.
      cyberslammer2
      • RE: Microsoft shares details of holiday Hotmail outage

        [i]@NonZealot Yet you have an iPhone.[/i]

        And after all his ridiculous tirades, do you really believe that?

        lol...
        ahh so
      • RE: Microsoft shares details of holiday Hotmail outage

        @cyberslammer2

        Now, now, I'm sure he's just waiting for his contract to run out so he can upgrade to WP7 ;-)
        tonymcs@...
    • RE: Microsoft shares details of holiday Hotmail outage

      @NonZealot

      LOL
      mikroland
  • RE: Microsoft shares details of holiday Hotmail outage

    sd
    peter238
  • RE: Microsoft shares details of holiday Hotmail outage

    Hi.
    Microsoft Hotmail hiccuped around the New Year holiday, resulting in approximately 17000 users finding some or all of their messages missing.
    .........
    <a href="http://www.gov-auctions.org" rel="dofollow">auto auctions</a>
    peter238
  • Poor Microsoft software.

    Not that surprising at all - Most things MS makes are poor and junk.
    itguy08
    • RE: Microsoft shares details of holiday Hotmail outage

      @itguy08
      Sad to say but you are so absolutely correct.
      Probably they are running the Hotmail service on Windows (cough, COUGH!) '"servers"'... Nuff said.
      Bob Wya
      • RE: Microsoft shares details of holiday Hotmail outage

        @Bob Wya They used to run Hotmail on Sun servers...wonder if they moved them to Server 2008.
        cyberslammer2
  • RE: Microsoft shares details of holiday Hotmail outage

    That is a really small number of people who were affected. Think of the millions and millions of users who use hotmail every day and only 17,000 had their email moved. I gotta give props to the hotmail team for their quick response and root cause analysis. That is the way it should be handled.
    Loverock Davidson
    • RE: Microsoft shares details of holiday Hotmail outage

      @Loverock Davidson Yep, and if it had been Apple you would have been laughing and screaming at them for being incompetent idiots.

      You're a true moron. I hope to meet you someday so I can shake the hand of a true idiot.

      Then rip your arm off and beat you over the head with it.
      cyberslammer2
      • RE: Microsoft shares details of holiday Hotmail outage

        [i]Then rip your arm off and beat you over the head with it.[/i]

        You'll have to rip off his safety helmet first. Then it might sink in.

        lol...
        ahh so
      • RE: Microsoft shares details of holiday Hotmail outage

        @cyberslammer2 <br><br>Oooooh new icon! Hope cyberspammer2 notices - that is f a g in the icon right?<img border="0" src="http://www.cnet.com/i/mb/emoticons/wink.gif" alt="wink"> <br><br>So now calling someone names and threatening violence is all you can up with?
        tonymcs@...
    • RE: Microsoft shares details of holiday Hotmail outage

      @Loverock Davidson
      You would be jumping up and down, yelling and screaming if you where one of those 17k people.
      It should never have happened.
      choyongpil
    • RE: Microsoft shares details of holiday Hotmail outage

      @Loverock Davidson

      Nope still broken. "Fixing" = ignoring
      do_not