ie8 fix
madison

Database glitch causes Windows 7 download server meltdown

By | April 30, 2009, 11:23am PDT

Summary: This morning, MSDN and TechNet subscribers were dismayed to find that downloads of the Windows 7 Release Candidate began bogging down shortly after they were made available. For several hours after the official launch, most subscribers who tried to log on found themselves unable to reach the download pages. The problem, I’m told by a Microsoft insider, wasn’t server capacity. Instead, the glitch (now fixed) was caused by a database configuration problem. I’ve got details and a startling graph.

When Microsoft released the Windows 7 beta for public download in January, the resulting demand overwhelmed its servers, delaying the launch by a day and giving the software giant’s capacity planners a black eye.

This morning at 6AM PDT, when Windows 7 Release Candidate downloads were officially made available for MSDN and TechNet subscribers, it looked like a sequel to that botched release. After 20 minutes or so of smooth downloads, both sites began bogging down, and the situation deteriorated rapidly as the minutes passed. For several hours after the official launch, most subscribers who tried to log on found themselves unable to reach the download pages.

This time, though, the problem wasn’t capacity. Instead, a source tells me, the glitch was caused by a SQL Server database that reached excessive fragmentation levels because of the tremendous surge of queries. How massive was the demand surge? The number of requests to the MSDN and TechNet databases in less than an hour was equal to more than a week’s traffic under normal circumstances.

The following graphic is from an internal Microsoft document explaining what happened. The blue line indicates percentage of processor usage, which is directly tied to fragmentation of the SQL Server database:

After the SQL Server index was rebuilt (just after 9:30AM), processor use dropped back to high but acceptable levels. I’m told that Microsoft engineers are now monitoring the status of this database every 30 minutes and plan to rebuild the indexes every evening to avoid a recurrence of the problem.

Reached for comment, a Microsoft spokesperson told me, “Due to high volume of traffic on the MSDN and TechNet sites this morning, many people may have experienced difficulties trying to download the Windows 7 Release Candidate. Microsoft has made changes to accommodate the increased traffic and subscribers shouldn’t experience any further issues.”

The good news for anyone awaiting the public download next Tuesday (May 5) is that those pages are tied to the same sort of subscriber database as the ones on MSDN and TechNet. Still, you can bet that an army of engineers will be watching that surge of traffic and wondering whether the third time is the charm.

Kick off your day with ZDNet's daily e-mail newsletter. It's the freshest tech news and opinion, served hot. Get it.

Ed Bott is an award-winning technology writer with more than two decades' experience writing for mainstream media outlets and online publications.

Disclosure

Ed Bott

Ed Bott is a freelance technical journalist and book author. All work that Ed does is on a contractual basis.

Since 1994, Ed has written more than 25 books about Microsoft Windows and Office. Along with various co-authors, Ed is completely responsible for the content of the books he writes. As a key part of his contractual relationship with publishers, he gives them permission to print and distribute the content he writes and to pay him a royalty based on the actual sales of those books. Ed's books written prior to fall 2011 have been distributed by Que Publishing (a division of Pearson Education) and by Microsoft Press. As of November 2011, Ed is a partner in the independent publishing company Fair Trade Digital Exchange, which exclusively publishes his books.

On occasion, Ed accepts consulting assignments. In recent years, he has worked as an expert witness in cases where his experience and knowledge of Microsoft and Microsoft Windows have been useful. In each such case, his compensation is on an hourly basis, and he is hired as a witness, not an advocate.

Ed does not own stock or have any other financial interest in Microsoft or any other software company. He owns 500 shares of stock in EMC Corporation, which was purchased before the company's acquisition of VMware. In addition, he owns 350 shares of stock in Intel Corporation, purchased more than two years ago. All stocks are held in retirement accounts for long-term growth.

Ed does not accept gifts from companies he covers. All hardware products he writes about are purchased with his own funds or are review units covered under formal loan agreements and are returned after the review is complete.

Biography

Ed Bott

Ed Bott is an award-winning technology writer with more than two decades' experience writing for mainstream media outlets and online publications. He's served as editor of the U.S. edition of PC Computing and managing editor of PC World; both publications had monthly paid circulation in excess of 1 million during his tenure. He is the author of more than 25 books on Microsoft Windows and Office, including the recently released Windows 7 Inside Out.

Related Discussions on TechRepublic

Did you know you can take part in these discussions with your ZDNet membership?
31
Comments

Join the conversation!

Just In

RE: Database glitch causes Windows 7 download server meltdown
beijing2008 14th Sep
Thanks everyone for the scanlation. I have that one too and it's so nice to get a translation. happy Hermes Wallet
0 Votes
+ -
M$ should have used LAMP
Linux Geek 30th Apr 2009
at least nobody knew what kind of system was serving windoze.
0 Votes
+ -
RE: M$ should have used LAMP
Heathen89 1st May 2009
I am sure they would have used LAMP if they were distributing a Linux distro... That way they would have only needed capacity for 15-20 downloads...
0 Votes
+ -
Haha !
alupin 4th May 2009
Haha are ya working for sells at Microsoft?
Thanks everyone for the scanlation. I have that one too and it's so nice to get a translation. happy Hermes Wallet
0 Votes
+ -
Ha HA haaaa !!!! happy
That's exactly wut I though when I saw that article...
Bill should've used a real system to dispatch his candys...
Should've used linux grin
0 Votes
+ -
Either way, that number of inquiries into Windows 7 tells you something about the OS. It is THE OS that people want.
0 Votes
+ -
Claiming a crash as a victory
InAction Man 30th Apr 2009
now that's new!
0 Votes
+ -
demand
blackhawk556 30th Apr 2009
well at least there is demand for the RC, if nobody was downloading it you would say something like this, "Hahaha nobody is downloading the OS because it sucks"

but it is obvious people DO WANT to try windows and DO like it
0 Votes
+ -
It's still a crash
InAction Man 30th Apr 2009
a non event in the windows world.
  • Flagged
0 Votes
+ -
you're right
blackhawk556 Updated - 30th Apr 2009
It is a crash I never said it wasn't. They should have been ready for this since January but they screwed up AGAIN. My point was that no matter what they do people will always hate them, even if they do make a good product.
0 Votes
+ -
I think it is intentional on M$ part
xXSpeedzXx 30th Apr 2009
trying to build hype. This is what the 5th release they have been caught bearing aZZ. Technical issue my foot.
0 Votes
+ -
Ed What's up with this?
Johnny Vegas 30th Apr 2009
Can you give more details of this "fragmentation"? Presumably these download sites only query the SQL database in readonly fashion. High cpu I can see since it's looking up all these MSDN subscriber accounts and with everyone probably there for only one thing (W7RC) each one is a cache miss. Still I would think even a single SQL instance could handle thousands of queries/sec for this kind of scenario which leads me to wonder just what kind of load are they seeing?
0 Votes
+ -
Contributr
Probably can't get any more details
Ed Bott 30th Apr 2009
The actual numbers are about as close to state secrets as you can get in Redmond.
0 Votes
+ -
Perhaps...
fairportfan 1st May 2009
...the SQL database in question is also listing data about downloads, etc ... in which case many records would be being rewritten with more data added, which is the recipe for fragmentation.
Why don;t they use oracle to handle to database part?
0 Votes
+ -
or PostgreSQL (nt)
brokndodge@... 30th Apr 2009
.
0 Votes
+ -
They must be using MS SQL for their database. What did you expect? (NT)
No More Microsoft Software Ever! 30th Apr 2009
NT
0 Votes
+ -
With M$'s Marketers outnumbering coders 8 to 1 what could you expect?
0 Votes
+ -
It's nice that they have resolved the download page glitch, but that doesn't resolve the real problem. I'm currently watching "Microsoft File Transfer Manager" running at ~ 90KB/s. That's less than 10% of my internet connections bandwidth.

Why hasn't Microsoft caught on to the fact that there are much better solutions out there. I'm sure they see massive demand waves like this every time they release something new. Blizzard found a great solution to this problem, years ago, when they built bittorrent into their downloader.

I don't usually send whiny messages, but it appears that I have some free time while I wait for my Windows 7 image to download over the next 9 hours. It's very tempting to just go out and grab the leaked image off the many bittorrent sites, but that would be ... never mind. I think that is exactly what I am going to do.
0 Votes
+ -
Simple answer
frgough 30th Apr 2009
Microsoft makes and sells enterprise-level software. It would not be so
good for sales if they had to admit their offerings can't handle
enterprise-level loads.
0 Votes
+ -
It gets worse
pravusnex@... 30th Apr 2009
Well then I'm here to tell you, they can't handle the loads. I'm noticing that my connection keeps dropping. I have had to tweak my downloader settings and babysit the connection to keep it going.
  • Flagged
0 Votes
+ -
I didn't say they could
frgough 1st May 2009
I said they have to act like they can.
0 Votes
+ -
Because Comcast kills it..
JT82 1st May 2009
they destroy blizzards P2P downloader. Instead most subs will just HTTP direct download it from various third-party sites and get it much quicker.
0 Votes
+ -
Baby sit the server farm?
Info-Dave 30th Apr 2009
Rebuild indexes?

While users wait hours to download and have to babysit connections?

I don't get it.
0 Votes
+ -
Windows installment sucks
Randalllind Updated - 30th Apr 2009
30 mins to transfer files after upgrading holy crap! Then another 5 to perpare my desktop. This really crappy compare to installing Vista.
0 Votes
+ -
BEE Honest
not of this world 30th Apr 2009
vista take a good 45 min on the fastest maCHINE

everybody knoes that installer is time
delimited....

50MIn to restore boot >>> Ha Ha Ha fool
0 Votes
+ -
I mean at the end
Randalllind 30th Apr 2009
It took a an hour to inatall Windows 7 upgrade mode. Not sure why the beta was faster compare to RC 1. Now that it is installed it's ok.

The last part transfer files took the longest. I am holding off installing on my laptop because the beta refuse to detect any drivers etc.

0 Votes
+ -
At least that's what I remember reading.
0 Votes
+ -
They should "leak" it again
InAction Man 2nd May 2009
and avoid many reliability problems.
I've just posted a technical explanation of what probably happened as far as SQL is concerned - see http://www.sqlskills.com/BLOGS/PAUL/post/Why-did-the-Windows-7-RC-failure-happen.aspx

Join the conversation!

Formatting +
BB Codes - Note: HTML is not supported in forums
  • [b] Bold [/b]
  • [i] Italic [/i]
  • [u] Underline [/u]
  • [s] Strikethrough [/s]
  • [q] "Quote" [/q]
  • [ol][*] 1. Ordered List [/ol]
  • [ul][*] · Unordered List [/ul]
  • [pre] Preformat [/pre]
  • [quote] "Blockquote" [/quote]
ie8 fix
ie8 fix

The best of ZDNet, delivered

ZDNet Newsletters

Get the best of ZDNet delivered straight to your inbox

Facebook Activity

White Papers, Webcasts, & Resources
ie8 fix
ie8 fix