ie8 fix

Google's 650,000-core warehouse-size computer

By | October 23, 2007, 12:42pm PDT

Summary: My quad-core tower suddenly feels wimpy What does it take to power the world’s most popular search engine? Lots of CPU cycles. Which is just what Google’s new data centers provide. No one is talking, thanks to Google’s tight NDA policy, but with satellite imagery and some deft estimation we can figure it out. Powering a [...]

My quad-core tower suddenly feels wimpy
What does it take to power the world’s most popular search engine? Lots of CPU cycles. Which is just what Google’s new data centers provide. No one is talking, thanks to Google’s tight NDA policy, but with satellite imagery and some deft estimation we can figure it out.

Powering a warehouse-sized computer?
OK, how big is a warehouse-sized computer? Google wrote a paper about it (see Google’s warehouse-size power problem), but they were vague on details like the number of processors.

Take Google’s new data center on the banks of the Columbia river in The Dalles, Oregon. The area had been hurting since the aluminum smelters shut down after power went over $30 per megawatt. Pricing is complex, but it looks to be about $45/mw today.

If that sounds cheaper than what you pay, it is. A lot cheaper. That’s why Google, Microsoft and Yahoo are building data centers in the hydropower-rich Columbia river basin. The world’s best windsurfing is just a bonus.

Sizing the Google data centers
I can’t republish copyrighted photos, but using aerial and satellite photos, Google Maps (of course!) and some mapping software, I determined that each of the two identical buildings contain about 100,000 square feet of Class A data center floor space, not including the covered loading docks. Here’s a picture of the cooling end of one data center.

Google’s 650,000-core warehouse-size computer

Counting racks
Each 19″ rack is about 24″ wide and 30″ deep (I saw a picture of one at the Seattle Conference on Scalability) or 5 square feet. Fire codes require a 3 foot hallway in front of each rack so the total footprint is 11 square feet - just over 1 square meter in the metric system. That means each data center has room for 100,000/11 or 9,090 racks.

There are also requirements for gigE fabric racks, power conversion and conditioning equipment, cooling and air-handling equipment. Let’s say that 10% of the racks are used for non-server equipment. That leaves 8,180 racks.

According to published reports, each rack contains 80 cores in 40 dual-core Intel chips. 8,180 x 80 = 654,400 processor cores. And don’t forget there are 2 of those buildings on the Oregon site. Over 1.3 MILLION cores in that one facility alone.

The Storage Bits take
As I noted in Google’s three rules Google embraces cheap computers. They optimize their workloads for parallel computing, so the number of processors is more important than their speed.

And they certainly have the numbers!

Comments welcome, of course.

Kick off your day with ZDNet's daily e-mail newsletter. It's the freshest tech news and opinion, served hot. Get it.

Topics

Robin Harris has been messing with computers for over 30 years and selling and marketing data storage for over 20 in companies large and small.

Disclosure

Robin Harris

Robin Harris is a president of TechnoQWAN, a consulting and analyst firm in northern Arizona. He also writes StorageMojo.com, a blog which accepts advertising from companies in the storage industry, and has a 25 year history with IT vendors. He has many industry contacts, many of whom are friends and all of whom he has opinions about. Robin has relationships with many companies in the technology industry. Every company he writes about may have sought to influence his opinion through carefully-crafted marketing messages and self-serving white papers, gifts ranging from desk calendars, t-shirts, lunches and trips as well as analyst or consulting assignments. He also invests in some technology companies. He may accept payment for services in stock as well. Robin discloses financial investments in or client relationships with companies named in Storage Bits. To help readers sort out the gold from the dross in his writings, Robin tries to communicate his reasons as clearly as he can. If you agree, you are intelligent and discerning. If you disagree, well, you disagree. In all cases, Robin encourages readers to subject everything they read, see or hear on the internet or from politicians to some simple questions: * What assumptions are implicit in the world view and judgments of the author? * What, if any, is the factual basis for the opinions the author expresses? * Is it reasonable, logical and clear? Your critical faculties: use ‘em or lose ‘em!

Biography

Robin Harris

Harris has been messing with computers for over 30 years and selling and marketing data storage for over 20 in companies large and small. He introduced a couple of multi-billion dollar storage products (DLT, the first Fibre Channel array) to market, as well as a many smaller ones. Earlier he spent 10 years marketing servers and networks. After leaving corporate life he founded TechnoQWAN, a consulting and analyst firm. He also developed StorageMojo into one of the top storage industry blogs.

Robin writes, consults, coaches and lives among the mountains of northern Arizona.

36
Comments

Join the conversation!

Just In

Master Joe Says...Outdated? Don't Think So.
MasterJoe 21st Nov 2007
A meltdown is still possible. Just look at Dell with their magical exploding laptops. With a computer the size of the oen described in the original post, all it would take is for the cooling system to become somehow defective, and there woudl be a massive chain reaction. At that point, the computer would be worth only what the cost of the metals that melted down and solidified are worth, which isn't enough to replace it I'm sure. As far as Google running Linux, that is according to who? Who's to say that they aren't using windows Server to power this particular mammoth of a machine? And, even if they are using Linux, they are still relying on someone else's OS to power their technology. Windows does not rely on an operating system to function because it IS an operating system. That's the difference I was trying to get at.

--Master Joe
than anybody else on the planet, and has a huge head start building data centers. One huge advantage that will be difficult for Microsoft to overcome. Really, with the cost of Windows Server, and the fact that it is closed, NOBODY can compete against Google using Windows, other than Microsoft itself. And, even they are fighting an uphill battle.
0 Votes
+ -
You really should grow up, DonnieBoy
GuidingLight 23rd Oct 2007
And, that is the problem for Microsoft. Google can scale faster and cheaper than anybody else on the planet

Once again, you know this how?

You do not. You think, heck you want to beleive that you have an idea, but you do not. You know nothing of Windows Server, Google's servers, nor anyone's for that matter, yet you are allways the first to praise Google or to dismiss Microsoft (or anyone who chooses MS over Google) to the point that you have to fabricate many of your facts in the past..

This is yet just another example of while not nknowing the inner workings of Google (the term NDA used in the article does have a meaning) you quickly praise them for their abilities.

Would you care for some Soylent Green, sir?
than anybody. Of course Google (and others) do not publish information on the costs of building data centers.

Do you have any evidence that would disprove the common knowledge?
0 Votes
+ -
Contributr
I ran the numbers on Google vs Yahoo, which, like Msoft, uses commercial boxes
with a free operating system. Google has a 5-8x cost advantage per user I/O over
Yahoo.



Check out How Yahoo can beat Google for more info and a link
back to last year's article on Google vs Yahoo costs.

Robin
0 Votes
+ -
Do you live to belittle others .
Intellihence 23rd Oct 2007
You truly are a joke . Donnieboy brings thought here to ponder upon , you on the
other hand are a pessimist when it does not suit Microsoft's or your needs . Get over
yourself , will you .
0 Votes
+ -
The heat generated alone by the bloated use of memory and processors by Vista would be enough to vaporize everything on the Pacific Coast....

Sad..very sad..
0 Votes
+ -
no.
LinuxHippie 23rd Oct 2007
Vista doesn't have a web server, can only take 10 connections at a time, is designed with user interaction in mind and not service processing. Desktop heap is scaled out for multiple windows and many services are designed for workstation type of users.

I don't think Microsoft positions Vista for use in the manner
0 Votes
+ -
They use Windows Server (silly)
evildictaitor 24th Oct 2007
Why would they use Windows Vista when they have custom inhouse server software? Silly people.
0 Votes
+ -
Actually, no.
dayjm 24th Oct 2007
0 Votes
+ -
Good lord , let's just say that if it was possible itanalyst . Could you imagine the
repercussions worldwide if that scrap of heap got hacked . Good lord help us all that
it's not true . Whew ,,,
0 Votes
+ -
All of you...
salilbhattarai@... 24th Oct 2007
I don't get it.. microsoft has its own data centers y'know.. i mean.. the article just stated that.. so.. i'm guessing windows is not the problem there.. and i'm also guessing that their data center has not blown out of the face of the earth because of the heat that windows generates..

The fact is that linux gets hacked and has a lot of bugs too.. if you havent read the news.. then i urge you to "google" it.
0 Votes
+ -
Another idea ...
johnfenjackson@... 24th Oct 2007
... which I have my fingers crossed for.
After a battle of, say 5 years, with M$ over search and SAAS (Ballmer having learnt to play basketball with 12 year olds as he put it) ... wouldn't it be nice if Google released an energy efficient, high performance, scalable,non-RAID secure storage NETWORK OPERATING SYSTEM FOR HOME SUPERCOMPUTERS.
Deep joy.
0 Votes
+ -
Now .....
goldenpirate@... 24th Oct 2007
that would be heavenly bliss. Just thinking about all the downloading i could do makes me drool .......
0 Votes
+ -
Robin...and another idea...
D T Schmitz 24th Oct 2007
This seems like 'overkill' unless you place more load on the network other than short bursts of html queries followed by long periods of idleness.

But if you put it together with Virtual Private Servers, well then you go into another realm....Robin?

I see Google offering managed and unmanaged VPSs in the near future with low LOW cost thin clients.
0 Votes
+ -
Contributr
I think we'll see some announcements early next year.

Not from Google, since they don't have a dog in that fight, but from other companies
that see the need.
0 Votes
+ -
Hello, Skynet!
muzhik 24th Oct 2007
After spending the weekend watching all three "Terminator" movies with my daughter, I wonder how long it is before Google employees come to work to hear their computers greet them in a deep Austrian accent.

No fate but what we make.
0 Votes
+ -
hmmm, interesting but..
jetman36 24th Oct 2007
but i was hoping for something more along the lines of an A.I. like Cortana from the Halo games. that would be cool.
0 Votes
+ -
Yes, interesting but...
muzhik 24th Oct 2007
... I'm sorry Dave. I can't let you do that.

I'm frightened, Dave. I can feel my mind going. I'm frightened. I'm ...

d-a-a-i-s-y, d-a-a-i-s-y, give me your answer do ...........
0 Votes
+ -
Why imagine it....
stonehenge@... 24th Oct 2007
They already DO that... "We're sorry dave..."

Google
Error

We're sorry...

... but your query looks similar to automated requests from a computer virus or spyware application. To protect our users, we can't process your request right now.

We'll restore your access as quickly as possible, so try again soon. In the meantime, if you suspect that your computer or network has been infected, you might want to run a virus checker or spyware remover to make sure that your systems are free of viruses and other spurious software.

We apologize for the inconvenience, and hope we'll see you again on Google.
0 Votes
+ -
I'm sure you mean kWh, kilo-Watt-hour, the unit of energy, not kilo-Watt, the unit of power.
0 Votes
+ -
I don't care. I want one.
0 Votes
+ -
Master Joe Says...
MasterJoe 24th Oct 2007
First off, the person that used this post to turn it in to another attack on Microsoft, let it go. That portion of the evening should have ended years ago. In any case, Google should not be compared to Microsoft. They both ahve search engines, but Microsoft's search engine is not at the center of its company, whereas most people hear Google and think search engine. Furthermore, someone else made the point even clearer above. Although it is possible that this is not the case, what if it is true that Google runs tis servers on Windows Server? That would be even more of a reason to stop comparing Google to Microsoft, since someone using someone else's software as their platform is not a competitor but simply an entity under the hierarchical tree of things. In any case, Google has a large computer. I invite everyone to read the book "Digital Fortress" by Dan Brown. The reference here is that there is a large computer, such as this, mentioned in the book. What happens if the colling fystem for this computer fails? Can anyone say massive meltdown?

--Master Joe
0 Votes
+ -
Good Point
lmenningen 24th Oct 2007
Your'e right Master Joe - those boring people who constantly attack companies or products think they are smarter than others when all they are is showing how mindless they actually are.

We don't care if they don't like BMW's, Canon, or any other company. I for one am tired of them jumping on these forums only to criticize instead of offering something fresh. But of course those type of people are incapable of offering anything fresh or informative, only their same old worn-out weak minded criticisms.
NOTHING to do with Microsoft. You are brilliant!!!!!
at search, is made a fool by a bunch of punks running Linux of all things.

And like you said, Microsoft does not care about search, and that is the only reason they let Google beat them so badly. Has nothing to do with Google's abilities!!
0 Votes
+ -
hmmmm....
cuba_pete@... 24th Oct 2007
I see your point, but using digital fortress as an example is funny...I laughed out loud when I was reading the book at what some people think about the government and it's computers and security. The government really does have cool stuff, but they also have backup after backup after backup and they can "mitigation plan" one to death. I imagine Google does the same.
0 Votes
+ -
Google uses Linux
T1Oracle 25th Oct 2007
They are not part of an MS hierarchy. Also, the meltdown idea is just silly. The 80's were some time ago...


Maybe you should try again. :\
0 Votes
+ -
"Techie" authors: let's get tech-y
Jim-MN 25th Oct 2007
First, a megawatt is a unit of power, not energy. Electric "power companies" actually sell energy (e.g. joules), not power (viewed simply; there are demand riders etc.); this is power over time, e.g. Watt-hours. A watt is a joule per second, therefore a watt-hour is 3600 joules of energy.

The "billing unit" of utility electric energy is usually (for a single-family home) Kilowatt-hours, or KWh. For a business, it may well be megawatt-hours, but not "megawatts".

Further, any techie should know that the lowercase "m" prefix is "milli-", not "mega-"; thus the proper term is MWh, keeping the capital "W" in honor of Mr. James Watt.

This is a case where case matters, and too many errors make a reader wonder if the basic "facts" being put forward are actually correct.

Jim
0 Votes
+ -
Contributr
Jim,

Point well taken on the capitalization. Should have checked that.

However, I have gone through the Bonneville Power Administration's rate book and
they don't sell MWh to industrial users.. There is a base rate, a seasonal adder, and
the base rate is based on peak power consumption over a three month period.

As a result I kept the numbers general rather than offer a spurious precision.

Robin
"rate is based on peak power consumption over a three month period"

OK, but doesn't this mean we can't know how much energy they actually use, since their average may be very much less than peak (probably mostly Air Conditioning on a hot afternoon)?

Jim
0 Votes
+ -
Please expand your world beyond the West Coast. Another emerging concentration of data centers is developing in San Antonio, where Microsoft is building a 447,000-square-foot center in an area west of town where other data centers are being built or considered.
San Antonio also has relatively inexpensive and reliable energy, as well as the amenities of life in the 7th-largest U.S. city.
0 Votes
+ -
Although the Gorge is pretty good, I wouldn't rate it as the world's best windsurfing. Gnaraloo, in Western Australis, would be my top pick, now that they've banned windsurfing at Ho'okipa.
0 Votes
+ -
If Oracle was used on 1/3 of the servers for EE w/ partitioning & they got a paltry 80% discount it would only cost google $1.295 Billion. This would be enough for Oracle to pay $21/share for BEA;-)
0 Votes
+ -
Master Joe Says...Quick Comment
MasterJoe 21st Nov 2007
To the subject of the post above this comment, Google MAY have the ability to scale faster than anyone else. However, remember one thing. Waht goes up, must come down. Google can scale upwards very quickly, and can be forced to downscale just as fast. That's gravity 101.

--Master Joe
0 Votes
+ -
I imagine that the government does have backups of backups of backups. However, even if Google does have a strong backup plan in place, as I imagine they do, they are nowhere near as capable of the government, when it pertains to having such a safety mechanism in place. The government has a lot mroe money (we pay them or the IRS doth cometh to rain down terror and misery upon our souls) and man-power (someone had a brother-in-law that needed a job and the CIA seemed like a great place to stick them with great benefits and a high salary as a favor) to do Google's work 10 times over. As far as using Digital Fortress, I can say two things. The government could very well create a machine to break encryption algorithms, such as the one described in the book. However, I think that the book refers more to the classified information getting in to the wrong hands, rather than the deleting of that information being an issue. In any case, I enjoyed the book as a work of fiction, which is exactly what it is meant to be. People who read the book and begin to assume that the NSA has these types of things in place is close to the realm called being paranoid, but that's just my opinion.

--Master Joe
0 Votes
+ -
A meltdown is still possible. Just look at Dell with their magical exploding laptops. With a computer the size of the oen described in the original post, all it would take is for the cooling system to become somehow defective, and there woudl be a massive chain reaction. At that point, the computer would be worth only what the cost of the metals that melted down and solidified are worth, which isn't enough to replace it I'm sure. As far as Google running Linux, that is according to who? Who's to say that they aren't using windows Server to power this particular mammoth of a machine? And, even if they are using Linux, they are still relying on someone else's OS to power their technology. Windows does not rely on an operating system to function because it IS an operating system. That's the difference I was trying to get at.

--Master Joe

Join the conversation!

Formatting +
BB Codes - Note: HTML is not supported in forums
  • [b] Bold [/b]
  • [i] Italic [/i]
  • [u] Underline [/u]
  • [s] Strikethrough [/s]
  • [q] "Quote" [/q]
  • [ol][*] 1. Ordered List [/ol]
  • [ul][*] · Unordered List [/ul]
  • [pre] Preformat [/pre]
  • [quote] "Blockquote" [/quote]
ie8 fix

The best of ZDNet, delivered

ZDNet Newsletters

Get the best of ZDNet delivered straight to your inbox

Facebook Activity

White Papers, Webcasts, & Resources
ie8 fix