Inside Hotmail's ops center
Worth reading: ACM Queue has an interview with Phil Smoot, who manages Microsoft's MSN/Hotmail communications platform. Smoot provides some insight into how Microsoft manages a large scale Web service using its own software and about the challenges in managing millions of mailboxes. He is interview by Ben Fried, managing director of Morgan Stanley's IT department.
From the article:
In the landscape of today’s megaservices, Hotmail just might be Mount Everest. One of the oldest free Web e-mail services, Hotmail relies on more than 10,000 servers spread around the globe to process billions of e-mail transactions per day. What’s interesting is that despite this enormous amount of traffic, Hotmail relies on less than 100 system administrators to manage it all.
The big thing you think about is cost. How much is this new feature going to cost? A penny per user over hundreds of millions of users gets expensive fast. Migration is something you spend more time thinking about over lots of servers versus a few servers. For example, migrating terabytes worth of data takes a long time and involves complex capacity planning and data-center floor and power consumption issues. You also do more up-front planning around how to go backwards if the new version fails.
Another big difference that I’ve found in shipping products versus shipping services is that you have to have a real awareness of exactly what effect an error or failure is going to have on the operations team. You have to ask: Is this really going to pull an engineer out of bed in the middle of the night?
Phil Windley also has a post on the Smoot interview.